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Abstract 


A  workshop  for  high  maturity  organizations  was  held  on  November  16-18,  1999,  at  the  Soft¬ 
ware  Engineering  Institute  (SEI)  in  Pittsburgh.  The  purpose  of  this  workshop  was  to  better 
understand  practices  that  characterize  Level  4  and  5  organizations.  Topics  of  discussion  in¬ 
cluded  both  practices  described  in  the  CMM  (Capability  Maturity  Model)  and  other  practices 
that  have  a  significant  impact  in  mature  organizations.  Two  themes  were  anticipated  to  be 
important  to  the  workshop  participants:  statistical  process  control  for  software  and  the  reli¬ 
ability  and  credibility  of  Level  4  and  5  assessments.  Additional  topics  were  solicited  from 
the  participants  on  CMM  integration,  measurement,  technology,  human  issues,  and  quality 
assurance.  This  report  contains  brief  summaries  of  the  high  maturity  organizations  partici¬ 
pating  in  the  workshop  and  the  various  working  group  reports. 
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XI 


1  The  November  1999  High  Maturity 
Workshop 


A  workshop  for  high  maturity  organizations  was  held  on  November  16-18,  1999,  at  the  Soft¬ 
ware  Engineering  Institute  (SEI)  in  Pittsburgh.  The  purpose  of  this  workshop  was  to  better 
understand  practices  that  characterize  Level  4  and  5  organizations.  This  workshop  was  by 
invitation  only.  The  SEI  invited  representatives  from  all  known  Level  4  and  5  organizations, 
Lead  Assessors  who  had  reported  assessing  a  Level  4  or  5  organization,  and  selected  indi¬ 
viduals  doing  Capability  Maturity  Model®  (CMM®)-related  high  maturity  work. 

Topics  of  discussion  included  both  practices  described  in  the  CMM  and  other  practices  that 
have  a  significant  impact  in  mature  organizations.  A  survey  on  high  maturity  practices  was 
distributed  in  October  1999  to  all  known  Level  4  and  5  organizations.  Participants  in  the 
workshop  were  briefed  on  the  preliminary  results  of  that  survey  to  inspire  discussion  within 
the  working  groups.  For  representatives  from  Level  4  and  5  organizations,  filling  out  the 
survey  was  a  prerequisite  for  attending  the  workshop. 

Two  themes  were  anticipated  to  be  important  to  the  workshop  participants:  statistical  process 
control  for  software  and  the  reliability  and  credibility  of  Level  4  and  5  assessments.  Addi¬ 
tional  topics  were  solicited  from  the  participants  prior  to  the  workshop  and  used  to  drive  the 
creation  of  working  groups. 

The  workshop  opened  with  brief  presentations  by  the  Level  4  and  5  organizations.  Repre¬ 
sentatives  from  26  high  maturity  organizations  participated  and  are  listed  in  Appendix  B.  A 
summary  of  their  presentations  is  provided  in  Section  2  of  this  report.  The  organizational 
summaries  were  followed  by  the  preliminary  results  of  a  survey  on  high  maturity  practices, 
briefly  discussed  in  Section  3.  The  workshop  participants  then  broke  into  working  groups, 
and  their  discussions  are  summarized  in  Section  4. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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2  Summaries  from  the  Level  4  and  5 
Organizations 


Representatives  from  Level  4  or  5  organizations  were  asked  to  make  a  brief  ten-minute  pres¬ 
entation  about  their  organization.  Information  requested  included 

•  the  organization  name  and  location 

•  a  point  of  contact  for  the  organization 

•  date(s)  assessed  at  what  maturity  level(s) 

•  Lead  Assessor(s) 

•  size  of  organization  (number  of  software  professionals) 

•  typical  range  of  program  size  (e.g.,  50-200  per  thousand  source  lines  of  code  [KSLOC]) 

•  primary  application  domain(s)  /  product  line(s) 

•  retum-on-investment  and  improvement  trend  data 

•  issues  in  achieving  Level  4  or  5 

•  unique  (distinguishing)  practices,  etc.,  that  might  contribute  to  the  workshop  discussions 
Summaries  supplied  by  the  organizations  are  provided  in  this  section. 


2.1  AlliedSignal  Defense  &  Space  Systems  (now 
Honeywell,  Avionics  Integration  Systems) 
Teterboro,  NJ 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 

Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


4 

November  1996 
Larry  Bramble 

Steve  Janiszewski,  stephen.janiszewski@honeywell.com 
Ellen  George,  ellen.george@honeywell.com 
www.6sigmasw.com 
40  software  engineers 
20K  to  200K  LOC 

Avionics,  vehicle  management  systems,  and  auto  test  systems 


Honeywell  AIS  is  a  100%  PSP/TSP  trained  and  operating  organization. 
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2.2  BFL  Software  Limited,  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


4 

June  1999 

Carolyn  Swanson 

Ms.  Sujatha  Balakrishnan 

General  Manager  (Quality  and  Training) 

sujatha.ravi@bflsoftware.com 

www.bflsoftware.com http://www.bflsoftware.com/ 

1000  plus  as  of  December  23,  1999 
Medium  sized  projects 

Development  of  systems  software,  applications  software,  ERP,  product 
development  in  areas  of  health  care  and  hospital  management 


2.2.1  ROI  and  Improvement  Trend  Data 

Cost  of  Quality  at  BFL  is  calculated  by  measuring  appraisal  cost  and  prevention  cost.  We  are 
continuously  on  the  sustaining  pitch  and  slowly  moving  towards  the  optimizing  level. 


2.2.2  Unique  or  Distinguishing  Practices 

Business  goals  and  Quality  goals  are  tightly  coupled.  Training/orientation  to  all  employees 
on  new/enhanced  processes  is  made  mandatory.  Process  evolution  on  down-to-up  approach 
through  SEPG  members  from  various  technical  teams  across  Business  Units.  Process  and 
Project  management  tools  customized  depending  on  technology  and  complexity  of  the  proj¬ 
ect  to  enhance  productivity  and  reduce  cycle  time  of  development.  Sharing  of  best  practices 
across  the  organization  through  a  central  repository  of  project  data.  Disseminating  knowl¬ 
edge  across  the  organization  through  SEPG,  Metrics  Council  and  Tools  Technology  and  Tran¬ 
sition  Group.  SQA  for  every  project  to  ensure  process  compliance  at  project  level  and  or¬ 
ganizational  level.  Quantifiable  quality  goals  for  each  project  and  phase-wise  analysis  to 
adopt  corrective  measures  to  achieve  the  defined  goals. 

2.2.3  Issues  in  Achieving  High  Maturity 

Convergence  of  various  tools  for  process  automation,  usage  of  statistical  data  for  process  im¬ 
provements  and  decision  making.  Changes  in  technology,  non-availability  of  information  on 
application  of  quantitative  process  management  to  software  business  realities.  No  clar¬ 
ity/guidelines  available  on  sampling  for  assessment  for  an  organization  spread  across  differ¬ 
ent  geographical  locations  and  executing  projects  on  various  domain  and  technology  areas. 
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2.3  The  Boeing  Company,  Space  Transportation 
Systems,  Kent,  WA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 


Point  of  Contact 


5 

July  1996 
Steve  Masters 
Mark  Paulk 
John  Vu 

Chuck  Martin,  Charles.Martin3@PSS.boeing.com 
Gary  Wigle,  gary.b.wigle@boeing.com 
Greg  Fulton,  gregory.p.fulton@boeing.com 


2.4  Citicorp  Information  Technology  Industries 
Limited  (CITIL),  Mumbai  and  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 


Point  of  Contact 


Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


5 

October  1999 
Ken  Dymond 
S.  Santhanakrishnan 
Anand  Kumar 
Anand  Kumar 
Head  -  SEPG 

Anand.Kumar@citicorp.com 

www.citil.com 

1000 

50-  1000 

Banking  products  and  services  -  retail,  corporate,  investment,  data 
warehouse,  internet 


The  CITIL  development  centers  in  Mumbai  and  Bangalore  were  assessed  at  Level  4  through  two  back- 
to-back  CBA  IPI  assessments  in  December  1995.  The  Lead  Assessors  were  Cynthia  Wise  and  Ken 
Dymond.  A  part  of  CITIL,  the  Data  Warehouse  Center  of  Excellence,  was  assessed  at  Level  5  in  a 
CBA  IPI  assessment  in  October  1999.  The  Lead  Assessors  in  this  assessment  were  Ken  Dymond,  S. 
Santhanakrishnan,  and  Anand  Kumar. 

2.4.1  ROI  and  Improvement  Trend  Data 


Mean  annual  growth  in  productivity  of  10  per  cent 


Mean  annual  reduction  in  defect  density  of  20  per  cent 


Percentage  rework  effort  halved  in  four  years 
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Price  of  non-conformance  down  by  33  per  cent  in  4  years 


ROI  of  100  %  in  the  first  year;  300  %  in  the  fifth  year 

2.4.2  Unique  or  Distinguishing  Practices 

Managing  continuous  inflow  of  requirements 

Interim  release  cycle  as  low  as  four  to  six  weeks 

Evolutionary  lifecycles  piloted,  evaluated,  and  found  effective 

Centralized-cum-dispersed  SEPG  responsibilities 

Planned  reuse  of  design  and  code  components  across  the  organization 

Software  rating  using  an  empirical  model 

Intranet  as  the  primary  communication  vehicle  for  process  improvement  activities  and  results 
e.g.,  process  changes,  process  pilots,  Process  Change  Control  Board 

Technology  evaluation  and  pilot 

Defect  prevention 

High  level  of  automation  of  software  engineering  and  management  activities 
End-to-end  automation  for  data  capture 

2.4.3  Issues  in  Achieving  High  Maturity 

Non-repetitive  nature  and  scope  of  work  coming  from  hundreds  of  customers 
High  growth  rate 

A  large  inflow  of  new  practitioners 

Reducing  life  spans  for  methodologies  and  technologies 

Business  pressures  to  improve  further  even  before  the  full  benefits  of  current  process  im¬ 
provement  initiatives  have  accrued 
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2.5  HCL  Perot  Systems,  Noida  and  Bangalore,  India 


Maturity  Level  5 

Date  of  Assessment  February  2000 

Lead  Assessor  Pradeep  Udhas 

Point  of  Contact  Rakesh  Soni,  rakesh.soni@hpsglobal.com 

Web  Page  www.hclperot.com 

Size  of  the  Organization  925 

Typical  Program  Size  0.5  Million  -  5  Million  LOC 

Primary  Application  Domain  Development 

Migration  and  re-engineering, -e-commerce  and  Web  development 

HCL  Perot  Systems  (HPS)  is  a  joint  venture  between  Perot  Systems  and  HCL  Technologies 
with  offices  in  USA,  Europe,  Asia  Pacific  and  India.  The  venture  was  set  up  to  strengthen 
Perot  Systems  Europe  &  US  software  delivery  capability  and  a  channel  to  market  in  Asia  Pa¬ 
cific  region. 

Enterprise-wide  ISO  9001  certification:  March  1998 
Enterprise-wide  CMM  Level  4  assessment:  July  1999 
Enterprise-wide  CMM  Level  5  assessment:  February  2000 
Future  Plans:  People  CMM  assessment  during  2001 

2.5.1  ROI  and  Improvement  Trend  Data 

HPS  closely  monitors  the  costs  related  to  the  quality  initiative.  It  has  achieved  ROI  of  400% 
on  the  investments  made  during  the  last  3  years.  This  covers  only  the  tangible  benefits  like 
improved  productivity,  lower  rework  efforts  and  bonus  received  for  meeting  the  quality  crite¬ 
ria  specified  by  the  customers. 

Defect  density  has  shown  improvement  ranging  from  5%  to  30%  across  various  lines  of 
business.  Productivity  has  shown  improvements  ranging  from  6%  to  80%. 

2.5.2  Unique  or  Distinguishing  Practices 

Commitment  by  senior  management  is  demonstrated  by  a  monthly  review  by  a  steering 
committee  headed  by  the  CEO. 

Independent  customer  satisfaction  surveys  are  performed  by  the  SEPG  These  surveys,  which 
have  provided  useful  inputs  for  process  improvement,  have  shown  continuously  improving 
trends. 
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There  is  organization-wide  participation  of  all  employees  in  process  improvement  activities. 
Rewards  and  recognition  are  provided  for  best  suggestions. 

There  is  mandatory  Quality  Management  System  (QMS)  training  at  the  time  of  induction. 
Retraining  is  provided  whenever  major  changes  are  carried  out  to  the  processes. 

Flexibility  of  the  QMS  and  extensive  tailoring  guidelines  support  adaptation  to  changing 
technologies. 

Best  practices  are  shared  across  the  organization. 

A  dedicated  competency  development  center  focuses  on  training  and  continuously  enhancing 
the  skills  of  the  employees. 

Monthly  internal  audits  are  performed. 

There  are  well-defined  processes  for  support  groups. 


2.5.3  Issues  in  Achieving  High  Maturity 

Issues  in  achieving  high  maturity  include: 


•  high  growth  rates 

•  varying  nature  of  projects 

•  reduced  cycle  time  for  Web  based  projects 


2.6  IBM  Global  Services  India,  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 

Size  of  the  Organization 
Primary  Application  Domain 


5 

Nov  1999 
Richard  Storch 

Asha  Goyal,  gasha@in.ibm.com 
Maya  Srihari,  smaya@in.ibm.com 
Over  1700  software  developers 

Software  Development,  Conversion 
Maintenance,  Product  Development 


Porting,  Re-engineering,  Testing, 


ISO  9001  accredited,  March  1995 

SEI CMM  Level  4,  achieved  in  November  1997 

SEI  CMM  Level  5,  achieved  in  November  1999 
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2.6.1  Unique  or  Distinguishing  Practices 

“Learnings”  (what  went  wrong,  what  went  well)  and  project  assets  are  systematically  gleaned 
from  projects,  categorized  for  easy  access  and  made  available  to  all  project  managers  who  are 
facilitated  during  Project  Initiation  and  Look-ahead  meetings  to  avoid/assimilate  practices. 

Tools  developed  for  internal  use  in  projects  are  evaluated  for  propagation  or  enhancement  in 
the  organization  by  forming  focus  groups. 

The  GQM  methodology  is  used  to  set  meaningful  goals  and  use  appropriate  measures  to  track 
them  for  different  types  of  projects. 

•  Very  detailed  and  simply  explained  metrics  guidelines  are  made  available  to  all  practitio¬ 
ners.  Performance  metrics  are  widely  communicated  and  best  practices  recognized.  Met¬ 
rics  experts  are  available  for  consultation.  Statistical  analysis  is  done  on  defects  and 
schedule  commitments  to  study  trends  and  common  causes  of  variation. 

Optimal  Automation  and  access  to  all  to  process  engineering  activities  via 

•  process  change  request  and  its  management 

•  process  release  management 

•  process  document  access  and  navigation 

Traceability  is  maintained  for  every  process  change  with  its  incorporation  in  standard  proc¬ 
esses  and  release  for  use.  It  is  valued  based  on  a  defined  mechanism  and  awards  given  for 
Best  Change  suggestion. 

High  focus  on  customer  satisfaction  is  achieved  through 

•  periodic  Customer  Satisfaction  Survey,  analysis  of  results  and  process  improvements  to 
enhance  services 

•  analysis  of  unsolicited  customer  feedback  for  process  improvements 

Business  Effectiveness  Surveys  are  conducted  to  ascertain  employee  satisfaction  /morale  and 
to  take  appropriate  actions  for  improvement. 

Quality  group  conducts  SQA  effectiveness  surveys  on  QA  group  interaction  with  projects, 
technology  usage  surveys  and  process  effectiveness  surveys  to  get  feedback  from  practitio¬ 
ners.  Learnings  database  is  continuously  enhanced.  New  process  assets  are  created  for  reuse. 

One  of  the  major  strengths  of  the  organization  is  the  Individual  Skill  Development  Program. 
This  is  created  by  individuals  in  consultation  with  their  managers  and  tracked  at  the  organi¬ 
zation  level  along  with  training,  and  supported  by  extensive  global  training  opportunities. 
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All  non-conformances  and  weaknesses  are  studied  for  the  root  causes;  specific  instances  are 
taken  up  for  organization  level  defect  prevention  by  training,  asset  creation  or  management 

focus. 

Defect  prevention  training  on  root  cause  analysis,  look-ahead  meetings  and  planning  guide¬ 
lines  on  triggers  etc.,  are  provided. 

Technology  changes  are  proactively  taken  up  and  processes  for  communication,  piloting, 
rollouts,  feedback,  transitioning  and  sharing  of  experiences  are  well  defined  and  supported  by 
full  time  resources.  Up-to-date  technology  for  day-to-day  activities  is  available  to  all  practi¬ 
tioners.  Access  to  worldwide  intranet  provides  vast  information  on  all  topics  in  addition  to 
access  to  Internet. 

Organization-wide  involvement  and  commitment  to  quality  is  visible  in  terms  of  management 
participation,  resources  availability  and  rewards,  and  a  focus  on  all  aspects  of  internal  and 
external  customer  satisfaction. 


2.7  Litton  Guidance  and  Control  Systems,  Woodland 
Hills,  CA 


Maturity  Level 

Date  of  Assessment 

Lead  Assessor 

Point  of  Contact 

Size  of  the  Organization 

Typical  Program  Size 

Primary  Application  Domain 


4 

December  1998 
Mark  Amaya 

Ray  Madachy,  madachyr@littongcs.com 
Approximately  125  software  professionals 
20  -  350  KSLOC 

Integrated  systems,  inertial  systems,  displays  and  controls,  IFF,  fiber 
optics  and  acoustic  sensors 


2.8  Lockheed  Martin  Management  &  Data  Systems, 
King  Of  Prussia,  PA 

Maturity  Level  4 

Date  of  Assessment  November  1998 

Lead  Assessor  Andy  Felschow 

Carol  Granger-Parker 
Dennis  Ring 

Point  of  Contact  M.  Lynn  Penn,  mary.lynn.penn@lmco.com 

Web  Pace  Must  enter  trough  LMCO.COM  then  go  to  M&DS 
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Size  of  the  Organization  about  4500 

Typical  Program  Size  500  KSLOC 

Primary  Application  Domain  Information  Systems 

2.8.1  ROI  and  Improvement  Trend  Data 

Defect  detection  has  moved  significantly  into  earlier  phases  of  the  life  cycle.  Reduction  of 
defects  went  from  14  per  KSLOC  to  10  per  KSLOC  (over  three  years). 

Expanded  SEPG  to  Process  Board  (cross  functional  and  cross  product  lines) — found  too 
much  integration  of  processes  not  getting  visibility  if  focused  only  on  software  processes. 


2.9  Lockheed  Martin  Mission  Systems,  Gaithersburg, 
MD 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 

Size  of  the  Organization 
Typical  Program  Size 


5 

October  1999 
Paul  Byrnes 

Paul  Weiler,  paul.weiler@lmco.com 
A1  Aldrich,  al.aldrich@lmco.com 

LM-MS  has  over  50  programs  with  approximately  3000  people  in¬ 
volved  in  software  development  and  maintenance. 

Program  team  size  ranges  up  to  300  people,  with  program  software 
size  up  to  5000K  source  lines  of  code. 


LM-MS  conducted  a  combined  assessment  using  two  different  models,  the  Capability 
Maturity  Model  (CMM)  for  Software,  Version  1.1,  and  the  EIA/IS  731.1  Systems 
Engineering  Capability  Model  (SECM). 


2.10  Lockheed  Martin  Naval  Electronics  and 
Surveillance  Systems  -  Manassas,  VA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 

Point  of  Contact 


5 

February  1999 
Judah  Mogilensky 
John  Travalent 
Donald  White 
Donald  G.  White 
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Member,  Integrated  Process  Group 
donald.g.white@lmco.com 

Program  Size  35K  to  1500K  Equivalent  Source  Statements 

Primary  Application  Domain  Embedded,  real-time  development  of  submarine  systems 


Lockheed  Martin  (LM)  Naval  Electronics  and  Surveillance  Systems  (NESS)  -  Manassas  or¬ 
ganization  was  assessed  at  Capability  Maturity  Model  (CMM)  Maturity  Level  5  (Optimizing) 
in  February,  1999.  This  achievement  was  the  culmination  of  over  20  years  of  continuous 
(software)  process  improvement  initiatives.  The  assessment  used  the  latest  CBA IPI  method¬ 
ology  and  was  led  by  Judah  Mogilensky  of  Process  Improvement  Partners,  Inc.  The  assess¬ 
ment  team  consisted  of  two  other  SEI  authorized  Lead  Assessors,  John  Travalent  and  Donald 
White. 

LM  NESS-Manassas  is  organized  by  business  area  and/or  program.  There  are  approximately 
240  software  engineering  professionals,  not  including  software  quality  engineers  or  software 
configuration  management  personnel.  The  software  domain  for  LM  NESS-Manassas  is  em¬ 
bedded,  real-time  development  of  submarine  systems.  These  applications  include:  sonar 
systems  and  trainers,  combat  systems  and  trainers,  navigation  and  trainers,  and  acoustic  sur¬ 
veillance.  LM  NESS-Manassas  has  developed  and  delivered  over  18  million  source  state¬ 
ments  within  the  last  20  years.  Program  sizes  typically  range  from  35,000  to  1,500,000 
Equivalent  Source  Statements  (ESS).1 

LM  NESS-Manassas  has  been  doing  process  improvement  activities  for  many  years.  Fagan- 
like  Peer  Reviews  were  first  introduced  in  the  software  engineering  discipline  in  the  late 
1970s  and  extended  to  the  system  engineering  discipline  in  the  mid-1980s.  The  first  process 
group  was  formed  and  began  documenting  an  organization  level  process  in  the  late  1980s. 
LM  NESS-Manassas  had  its  first  assessment  (using  the  Software  Process  Assessment  meth¬ 
odology),  in  September  1990.  This  assessment  resulted  in  a  Maturity  Level  3  rating.  Since 
that  time,  many  other  process  improvements  have  been  incorporated  into  the  organization 
level  process  and  periodic  reassessments  have  validated  the  increased  maturity  of  the  organi¬ 
zation.2  Additionally,  several  Software  Capability  Evaluation  (SCEsm)  appraisals  have  oc¬ 
curred,  but  with  no  indication  as  to  the  maturity  level  received. 


1  The  Equivalent  Source  Statement  (ESS)  is  used  to  estimate  labor  requirements.  A  normalized  number 
reduces  different  source  statement  (SS)  types  to  the  effort  that  would  be  required  in  terms  of  newly 
developed  SS.  The  default  coefficients  can  be  changed  based  on  program  considerations  (e.g.,  COTS 
coefficient  can— and  should— be  made  non-zero  if  non-trivial  effort  is  involved  in  its  incorporation, 
such  as  customizing,  configuring,  etc.) 
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October,  1992 
June,  1995 
February,  1999 


SPA  Methodology  Level  3 

CBA  IPI  Methodology  Level  4 

CBA  IPI  Methodology  Level  5 
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2.10.1  ROI  and  Improvement  Trend  Data 

LM  NESS-Manassas  has  continually  monitored  important  factors  that  indicate  the  success  or 
failure  of  the  process  improvement  activities.  One  of  these  indicators  is  delivered  product 
quality.  For  most  LM  NESS-Manassas  customers,  this  translates  into  defects-per-delivered 
source  statements.  (Other  factors  such  as  cost  and  schedule  performance  are  also  monitored.) 
Since  the  inception  of  continual  process  improvement,  the  defect  density  (defects  /  million 
delivered  source  statements)  has  dropped  several  orders  of  magnitude  while  continuing  to 
deliver  the  product  on  time  and  within  budget. 

2.10.2  Issues  in  Achieving  High  Maturity 

One  of  the  biggest  obstacles  LM  NESS-Manassas  faced  in  its  journey  to  Levels  4  and  5  was 
being  able  to  understand,  educate,  and  institutionalize  the  difference  between  “mature”  man¬ 
agement  based  on  measurements  and  Quantitative  Process  Management.  However,  many 
other  practices  that  have  been  in  place  for  quite  some  time  have  served  LM  NESS-Manassas 
well.  Examples  of  these  practices  are 

•  use  of  Latent  Defect  model(s) 

•  use  of  software  engineers  in  Software  Quality  Assurance  (SQA)  roles 

•  augmenting  the  CMM  with  ISO  9001 

•  SQA  audits  driven  by  ISO  elements  and  CMM  key  process  areas 

•  pre-tailoring  of  the  organization's  standard  software  process  (OSSP)  based  on  the  type  or 
size  of  the  program 


2.11  Lockheed  Martin  Federal  Systems,  Owego,  NY 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


5 

December  1997 

John  Travalent 

Warren  A.  Schwomeyer 

Senior  Quality  Engineer 

warren.schwomeyer@lmco.com 

560 

IK  to  300K  source  lines  of  code 
Avionics,  postal,  and  commercial 


Lockheed  Martin  Federal  Systems  (LMFS)  -  Owego  was  assessed  at  Level  5  of  the  Capabil¬ 
ity  Maturity  Model  for  Software  (CMM)  in  December  1997.  This  achievement  marked  a 
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significant  milestone  in  the  organization’s  long  history  of  continuous  quality  improvement. 
This  discussion  will  address  the  software  organization  and  its  demographics,  the  software 
process  improvement  history  of  the  organization,  and  the  challenges  we  see  ahead  of  us  for 
continuous  software  process  improvement. 

There  are  approximately  560  software  professionals  at  the  site.  The  primary  application  do¬ 
mains  are  Avionics  (including  mission,  sim/stim,  diagnostic  and  real-time  embedded  sys¬ 
tems),  Postal  (including  optical  character  recognition  and  material  handling)  and  commercial 
(including  infrastructure  modernization  and  medical).  Our  programs  range  in  size  from  IK  to 
300K  source  lines  of  code  with  some  outliers  outside  both  ends  of  this  spectrum. 

LMFS  -  Owego’s  software  process  journey  started  in  the  late  1970’s.  Process  development 
and  improvement  continued  through  the  1980s.  We  were  assessed  at  Level  2  (and  well  on  the 
way  to  Level  3)  in  1991  using  the  Software  Process  Assessment  (SPA)  methodology.  From 
1992  through  the  present  we  have  had  almost  annual  Software  Capability  Evaluations  per¬ 
formed  by  our  customers  as  an  acquisition  activity.  Each  of  these  was  performed  to  CMM 
Level  3  and  the  organization  satisfied  all  levels.  In  1996  the  organization  was  assessed  at 
CMM  Level  3  using  the  CMM  Based  Appraisal  for  Internal  Process  Improvement  (CBA/IPI) 
method.  Half  of  the  goals  at  both  Levels  4  and  5  were  satisfied  indicating  the  process  was 
Level  5  capable,  but  needing  a  longer  period  of  performance  to  demonstrate  institutionaliza¬ 
tion.  In  1997,  the  organization  achieved  CMM  Level  5  using  the  CBA/IPI  method  (Lead  As¬ 
sessor:  John  Travalent). 

2.1 1 .1  ROI  and  Improvement  Trend  Data 

The  two  key  measures  of  process  performance  have  increased  through  this  period  of  im¬ 
provement.  Productivity  (delivered  source  lines  of  code  per  labor  month)  has  increased  an 
average  of  11.1%  per  year  since  1982.  Post-delivery  defect  rate  (software  defects  per  million 
delivered  source  lines  of  code)  has  improved  an  average  of  11.7%  per  year  since  1992.  Dur¬ 
ing  this  period  and  while  improving  these  performance  measures,  the  organization’s  customer 
set  transitioned  from  more  than  90%  Department  of  Defense  (DoD)  to  a  balance  of  DoD, 
Postal  and  commercial. 

2.11.2  Issues  in  Achieving  High  Maturity 

Some  of  the  challenges  which  lie  ahead  include  the  application  of  the  process  set  across  an 
ever-widening  spectrum  of  program  types  (DoD  and  commercial)  and  sizes.  Commercial 
off-the-shelf  (COTS)  programs  also  present  unique  challenges  in  applying  the  process  set. 
Another  challenge  before  us  is  the  development  of  Statistical  Process  Control  methods  and 
tools  to  bring  this  tool  out  of  the  hands  of  the  statisticians  for  use  by  the  practitioners. 

LMFS-Owego  embraces  the  CMM  principles  and  practices  beyond  the  software  processes  by 
embedding  them  in  our  Integrated  Systems  Development  (ISD)  process  and  applying  them  to 
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our  process  management  as  well  as  program  management.  LMFS-Owego  is  looking  ahead  to 
maturing  the  System  Engineering  processes  and  CMMI. 


2.12  Motorola,  GSM  (Global  System  for  Mobile 
Communications)  Systems  Division,  Network 
Systems  Group,  Arlington  Heights,  IL 


Maturity  Level 

Date  of  Assessment 

Point  of  Contact 

Size  of  the  Organization 

Primary  Application  Domain 


5 

October  1997 

Barbara  Hirsh,  hirsh@cig.mot.com 
250  in  1997 
Cellular  infrastructure 


2.12.1  Unique  or  Distinguishing  Practices 

Orthogonal  Defect  Classification 

Statistical  correlation  of  customer  satisfaction  scores  to  internal  metrics 
QPM  training 

2.12.2  Issues  in  Achieving  High  Maturity 

Rapid  growth 


2.13  Motorola  India  Electronics  Ltd.  (MIEL), 
Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


5 

November  1993 
John  Pellegrin 

Sarala  Ravishankar,  sarala@miel.mot.com 
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2.14  NIIT  Limited,  New  Delhi,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 
Web  Page 

Size  of  the  Organization 

Typical  Program  Size 
Primary  Application  Domain 


5 

September  1999 
Richard  Storch 

Bhaskar  Chavali,  BhaskarC@NIIT.com 
www.niit.com 

NUT  has  3800  employees  across  all  business  groups.  At  the  time  of  the 
CMM  assessment  in  Sept  1999,  the  software  group  had  450  in  New 
Delhi,  250  in  other  metros,  300  abroad 

3000  FPs  (1000-  15000  FPs) 

e-commerce,  Web  applications,  client/server  applications,  re¬ 
engineering,  migration/conversion,  maintenance 


ISO  certifications  and  CMM  assessments: 


•  First  ISO  9001  certification  from  BVQI,  UK  in  1994 

•  ISO  9001  recertification  from  BVQI,  UK  in  1997 

•  Assessed  at  SEI  CMM  Level  3  in  July  1997 

•  Assessed  at  SEI  CMM  Level  5  in  Sept  1999 

2.14.1  Unique  or  Distinguishing  Practices 

NIIT  has  implemented  Crosby’s  Complete  Quality  Management  System  (CCQMS),  pro¬ 
pounded  by  the  quality  guru  Philip  Crosby.  The  Quality  Education  System  (QES)  is  based  on 
Crosby’s  four  absolutes  of  quality.  Training  for  QES  is  provided  to  all  employees  as  part  of 
the  organization  induction  program.  The  premise  of  the  Personal  Quality  Initiative  (PQI)  is 
that  if  each  individual  improves  in  personal  and  professional  areas  due  to  their  own  personal 
quality  initiatives,  then  the  organization  benefits  in  the  long  term.  This  methodology  enables 
each  NIIT  employee  to  improve  his  /  her  personal  quality  by  providing  the  opportunity, 
methodology  and  the  tools.  All  NUT  employees  have  been  trained  to  use  PQI. 

NIIT  has  a  dedicated  process  group  and  TQM  group  to  facilitate  the  maintenance  and  crea¬ 
tion  of  the  software  development  and  maintenance  processes.  Almost  all  the  functional  or¬ 
ganizations, -such  as  Financial  Service  Organization  (FSO),  Internal  Communication  Organi¬ 
zation  (ICO),  Commercial  Services  Organization  (CSO),  Human  Resource  Organization 
(HRO),  etc.,  within  NUT  are  ISO  certified. 


2.14.2  ROI  and  Improvement  Trend  Data 

From  April  1998  to  September  1999: 

•  14%  improvement  in  productivity 

•  33%  reduction  in  effort  overrun 
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•  31%  reduction  in  schedule  slippage 

•  36%  reduction  in  (rework/total  effort)  ratio 

•  20%  decrease  in  PONC/person  (Price  Of  Non  Conformance) 
Productivity  Index  of  27  in  September  1999  (calculated  through  SLIM). 


2.15  Northrop  Grumman,  Air  Combat  Systems, 
Integrated  Systems  and  Aeronautics  Sector,  El 
Segundo,  CA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 
Size  of  the  Organization 

Typical  Program  Size 

Typical  Program  Size 

Primary  Application  Domain 


4 

Oct  1998 
Don  Dortenzo 

Leitha  Purcell,  purcele@mail.northgrum.com 
Approximately  350  software 
professionals  at  time  of  assessment. 
40KSLOC  to  1MSLOC 
10  to  over  200  software  engineers 
40KSLOC  to  1MSLOC 
10  to  over  200  software  engineers 
Avionics  systems,  ground  support 
systems,  weapon  systems 


2.16  Oracle  Software  India  Limited,  India  Development 
Center,  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


4 

May  1999 
Pradeep  Udhas 

Ashish  Saigal,  asaigal@in.oracle.com 
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2.17  Raytheon  C3I  Fullerton  Integrated  Systems, 
Command  and  Control  Systems/Middle  East 
Operations,  Fullerton,  CA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessors 


Points  of  Contact 
Size  of  the  Organization 


Typical  Program  Size 


Primary  Application  Domain 


5 

Oct  1998 
Paul  Byrnes 
Jane  Moon 
Ronald  Ulrich 
Ivan  Flinn 
Bruce  Duncil 
Janet  Bratton 
Jim  Hudec 

Jane  A.  Moon,  jmoon@west.raytheon.com 
Janet  Bratton,  jabratton@west.raytheon.com 

Approximately  1400  people,  including  400  software  engineers,  450 
systems  and  hardware  engineers,  and  150  support  staff.  Approxi¬ 
mately  360  software  engineers  in  1998  at  time  of  assessment. 

60-220  software  engineers 
0.75-2  millions  lines  of  developed  source  code 
40-200  embedded  COTS  software  products 
Command,  control,  and  information  systems 
International  air  traffic  control  systems 
Traffic  management  systems 


2.18  Raytheon  Missile  Systems,  Software  Engineering 
Center,  Tucson,  AZ 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


Size  of  the  Organization 
Typical  Program  Size 


4 

October  1998 

John  Ryskowski 

Michael  D.  Scott 

Department  Manager 

mscottl  @  west.raytheon.com 

Approximately  800  software  professionals 

There  were  250  in  Oct  1998 

Team  sizes  range  from  2  to  30 


18 


CMU/SEI-2000-TR-003 


KSLOCs  from  10K  to  250K 


Primary  Application  Domain  Missile  systems 

The  Software  organization  at  Raytheon  Missile  Systems,  in  Tucson,  was  assessed  at  CMM 
Level  4  in  October  of  1998.  This  achievement  marked  a  significant  milestone  in  the  organi¬ 
zation’s  quest  for  continued  process  maturity  and  improvement.  This  discussion  will  address 
the  software  organization  and  its  demographics,  the  process  improvement  history  of  the  or¬ 
ganization,  our  return  on  investment,  and  the  issues  we  continue  to  face  as  we  move  strive  to 
move  forward. 

The  engineering  directorate  in  Tucson  is  organized  by  function  or  discipline.  There  are  ap¬ 
proximately  800  software  professionals,  which  include  software  quality  engineers  and  soft¬ 
ware  configuration  management  personnel.  This  compares  to  just  250  in  October  of  1998, 
over  300  percent  increase  in  just  over  a  year.  This  growth  has  been  a  significant  factor  in  our 
ability  to  maintain  our  process  improvement  momentum,  as  will  be  discussed  later.  The 
software  domain  for  Tucson  is  embedded  real-time  development  of  missile  systems  and  sup¬ 
port  software.  Our  team  sizes  range  from  2  to  30  with  program  sizes  from  10K  to  250K 
source  lines  of  code. 

Although  the  Missile  Systems  organization  had  been  doing  process  improvement  activities 
for  many  years  it  was  not  until  the  early  90’s  that  our  activities  began  to  focus  around  the  Ca¬ 
pability  Maturity  Model.  In  1994  we  baselined  our  process  capability  using  the  results  of  a 
Software  Capability  Evaluation  done  as  an  acquisition  activity  and  internal  evalua¬ 
tions.  We  estimated  our  capability  at  CMM  Level  2.  A  detailed  software  process  improve¬ 
ment  plan  was  developed  and  implemented  over  the  following  two  years,  and  in  1996  the 
organization  was  assessed  at  CMM  Level  3  using  the  CBA IPI  method.  Based  on  the  find¬ 
ings  of  the  1996  assessment  and  the  organization’s  goals,  a  new  software  process  improve¬ 
ment  plan  was  developed,  and  in  1998  the  organization  achieved  CMM  Level  4  (again  under 
the  CBA  IPI  method — Lead  assessor:  John  Ryskowski).  Our  future  goals  are  to  achieve 
Level  5  in  2001. 


2.18.1  ROI  and  Improvement  Trend  Data 

A  key  factor  in  our  success  has  been  the  ability  to  show  that  our  efforts  have  had  value  to  the 
overall  organization.  We  continue  to  look  at  our  return  on  investment  to  make  sure  that  we 
are  truly  making  the  progress  we  expect.  Our  latest  data  includes  the  investments  made  from 
1994  through  1998,  looking  at  software  quality  (reduction  of  rework  costs  through  defect 
containment),  productivity  (lines  of  code  per  hour  improvement),  and  customer  satisfaction 
(program  capture  rate).  Costs  incurred  included  both  organizational  costs  as  well  as  program 
costs  for  process  and  process  improvement  activities.  We  acknowledge  that  this  is  not  a  true 
experiment  and  that  there  are  many  other  factors  that  come  in  to  play  (e.g.,  better  tools,  more 
experienced  workforce,  etc.),  but  we  do  believe  that  our  process  activities  have  been  key  to 
our  6: 1  return  on  investment  over  this  four-year  period. 
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2.18.2  Issues  in  Achieving  High  Maturity 

We  still  have  a  long  way  to  go  to  achieve  our  ultimate  goal  of  continuous  process  improve¬ 
ment  as  a  way  of  life  in  the  organization.  Significant  issues  continue  to  place  challenges  in 
our  path.  In  the  last  year  Raytheon  has  consolidated  all  of  its  missile  business  to  Tucson 
which  has  more  than  tripled  the  number  of  programs  and  personnel  in  the  organization.  With 
this  growth  has  been  a  significant  influx  of  new  engineers.  We  continually  have  to  “sell” 
process  and  process  improvement  to  program  management  (“Why  do  we  need  to  be  higher 
than  Level  3?”).  And  finally,  there  continues  to  be  issues  regarding  process  funding,  espe¬ 
cially  with  other  disciplines  getting  on  board  (a  good  thing,  but  spreads  available  funding 
thin). 

Tucson  has  made  great  progress  over  the  past  six  years  and  we  expect  to  continue.  We  are 
well  on  our  way  to  Level  5  and  are  keeping  our  eye  on  the  next  challenge  of  CMMI. 


2.19  Satyam  Computer  Services  Ltd.,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


Level  5  in  for  all  Indian  locations 
March  1999 
Richard  Knudson 

Prabhuu  Sinha,  prabhuu@satyam.com 
www.satyam.com 

4000  total  employees,  3300  software  professionals 
10-100  KSLOC 

Satyam  is  a  complete  IT  solutions  provider. 


Satyam  was  ISO/TickIT  certified  in  March  1995,  and  re-certified  in  September  1998. 


The  nature  of  projects  done  at  Satyam  include 

•  maintenance 

•  new  development 

•  conversion 

•  migration 


The  types  of  projects  are  mainly 

•  monolithic  information  systems 

•  two-tier  client-server  software 

•  three-tier  client-server  software 
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•  systems  software 

•  embedded  software 

•  commercial  software  packages 

2.19.1  ROI  and  Improvement  Trend  Data 

Improvements  observed  in  past  one  year 
Development  Projects 

•  40  %  Reduction  in  Effort  Estimation  Variance 

•  24  %  Reduction  in  Schedule  Estimation  Variance 

•  8  %  Reduction  in  Delivered  Defect  Density 

Maintenance  Projects 

•  53  %  Reduction  in  Effort  Estimation  Variance 

•  34  %  Reduction  in  Schedule  Estimation  Variance 

•  50  %  Reduction  in  Delivered  Defect  Density 

2.19.2  Unique  or  Distinguishing  Practices 

Satyam  uses  an  enterprise  tool  for  monitoring 

•  Implementation  of  CMM 

•  Implementation  of  QMS 

Satyam  uses  a  tool  for  providing  organization-wide  access  to 

•  quality  system 

•  process  training 

•  project  knowledge  base 

•  process  suggestions  box 

All  business  units  and  support  units  in  Satyam  operate  on  a  Profit  Center  approach  with  an 
internal  customer-supplier  relationship.  All  support  units  also  have  defined  processes.  Every 
business  unit  /  support  unit  reviews  its  quality  issues  every  month.  SQA  conducts  regular 
project  process  monitoring  to  provide  continuous  process  implementation  support  to  projects. 
Projects  conduct  regular  project  introspective  /  retrospective  meetings  to  share  the  learning 
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from  the  project  with  other  Project  Leaders.  The  Satyam  Learning  Center  facilitates  the 
building  up  of  individual  and  organizational  skills  and  competencies. 

Satyam  has  dedicated  support  units  for  building  competencies  on 

•  tools  and  technologies 

•  business  domains 

2.1 9.3  Issues  in  Achieving  High  Maturity 

Quantifying  the  impact  of  varying  project  attributes  on  productivity  and  quality 

Satyam,  being  an  IT  solutions  provider,  conducts  different  types  of  projects  for  different 
customers.  Examples  of  varying  project  attributes  are  environment  details  (Platform,  lan¬ 
guage,  etc.),  project  specific  attributes  (Project  Size,  complexity,  domain,  etc.),  project  team 
skills  and  customer  specific  attributes  (how  they  present  requirements  and  changes,  their  own 
process  maturity  and  appreciation,  culture,  language,  schedules  imposed  by  them). 

Organization  level  productivity  and  quality  baselines  are  difficult  to  set  since  there  are  very 
few  similar  projects.  The  impact  is  both  in  terms  of  normalization  at  organization  level  and 
goal  setting  at  project  level. 

In-process  (mid-term)  corrective  actions  (QPM)  in  short  duration  projects  (three  to  four 
months)  are  not  very  effective.  They  are  also  not  easy  to  implement  due  to  high  schedule 
pressure.  Goal  setting  for  new  technology  projects  by  piloting  is  difficult  because  the  pilot 
projects  in  Satyam  are  very  small  in  size  and  may  not  give  feasible  data.  Defect  prevention 
learning  in  early  life-cycle  phases  often  cannot  be  used  within  the  same  project.  It  is  used  in 
other  projects  via  sharing.  Due  to  these  factors,  the  project  does  not  see  the  immediate  benefit 
of  causal  analysis  and  DP  meetings. 

2.20  United  Space  Alliance,  Space  Shuttle  Onboard 
Software  Project,  Houston,  TX 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


5 

Nov  1989 
Donald  Sova 

Julie  R.  Barnard,  julie.r.bamard@usahq.unitedspacealliance.com 

www.unitedspacealliance.com 

250 

400K  lines  of  avionics  software,  1.3M  lines  of  application  software 
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Primary  Application  Domain  Space  Shuttle  software 

The  Space  Shuttle  Onboard  Software  project  is  a  part  of  the  United  Space  Alliance  Corpora¬ 
tion  based  in  Houston,  Texas.  The  project  was  previously  operated  by  IBM,  Loral,  and 
Lockheed  Martin  before  joining  United  Space  Alliance  on  July  4,  1998.  The  project  (under 
IBM  at  the  time)  was  assessed  at  CMM  Level  5  in  November  of  1989  by  an  SEI-trained 
NASA  team,  led  by  Donald  Sova  of  NASA  Headquarters  Code  Q. 

The  Space  Shuttle’s  flight  software  (Primary  Avionics  Software  System,  PASS)  is  responsible 
for  the  guidance,  navigation,  and  flight  control  functions  performed  during  all  phases  of  the 
Shuttle  flight.  The  flight  software  consists  of  over  400,000  lines  of  code  that  run  on  the 
Shuttle’s  onboard  computers.  To  satisfy  NASA’s  requirement  for  software  that  meets  the 
highest  safety  and  reliability  standards,  the  PASS  flight  software  project  evolved  a  software 
process  that  yields  a  highly  predictable  quality  result.  By  executing  the  process  faithfully  to 
specified  process  standards,  the  flight  software  produced  by  the  process  is  predictably  near 
zero  defects. 

The  project  also  develops  and  maintains  over  1 .3  million  lines  of  Flight  Software  Application 
Tools  that  support  configuration  management,  software  builds,  test  and  simulation,  automatic 
verification,  and  software  reconfiguration.  The  rigorous  processes  applied  to  these  tools  have 
yielded  exceptional  product  quality  levels  as  well. 

The  project  size  is  generally  around  250  people.  The  project  resides  currently  within  a  larger 
software  organization  of  approximately  600  people,  which  includes  a  major  subcontractor 
assessed  at  CMM  Level  5  in  October  1999. 

2.20.1  Unique  or  Distinguishing  Practices 

Some  of  the  distinguishing  practices  the  project  is  recognized  for  include 

•  rigorous  application  of  formal  inspections,  including  software  requirements  inspections 

•  in-depth  defect  prevention  process,  including  the  detailed  examination  of  every  software 
defect  from  a  process  escape  perspective 

•  matrixed  program  management  and  control  board  structure 

•  long-term  continuous  process  improvement 


2.20.2  Issues  in  Achieving  High  Maturity 

One  of  the  challenges  facing  the  project  over  the  year  period  since  its  CMM  Level  5  assess¬ 
ment  has  been  to  sustain  its  practices  across  the  company  changes  and  reorganizations.  By 
leaving  intact  the  organization  structure,  core  management,  and  process  infrastructure,  the 
best  practices  continued  successfully  across  the  corporate  boundaries. 
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Future  activities  include  transferring  best  practices  to  other  parts  of  the  company.  The  re¬ 
cently  established  corporate  software  process  owner  and  corporate  SEPG  would  be  the  pri¬ 
mary  mechanism  for  advancing  the  software  process  maturity  of  the  overall  company  soft¬ 
ware  process. 


2.21  Tata  Elxsi  Limited  (TEL),  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


4 

August  1999 
Pradeep  Udhas 
M  Thangarajan 

Corporate  Manager  -  Quality  and  Training 

mtr@teil.soft.net 

www.tataelxsi.com 

300 

5K  to  150K  source  lines  of  code 

Networking  protocol  development  (TCP/IP,  ATM,  ISDN,  SNMP, 
BGP,  etc) 

System  development  (embedded  systems,  ASIC  design,  VHDL  and 
Verilog  modeling,  firmware  including  DSP) 

Visual  computing  (2D/3D  graphics,  animation,  data  visualization,  etc) 
Internet  &  intranet  group  (Web  enabling  of  products). 


The  Design  &  Development  Center  of  Tata  Elxsi  Limited,  in  Bangalore,  was  assessed  at 
CMM  Level  4  in  August  of  1999.  This  achievement  marked  a  significant  milestone  in  the 
organization’s  quest  for  continued  process  maturity  and  improvement. 

The  Quality  Management  System  of  our  organization  was  first  designed  to  meet  the  ISO 
9001,  Tick  It  guidelines,  and  we  obtained  the  certification  in  February  1997.  Subsequently 
our  processes  were  greatly  enhanced  to  meet  the  requirements  of  the  Capability  Maturity 
Model,  and  the  necessary  implementations  were  carried  out  over  the  next  two  years.  On 
August  13,  1999,  the  organization  was  assessed  at  Level  4  using  the  CBA-IPI  method.  The 
assessment  was  conducted  by  KPMG  (Lead  Assessor:  Mr.  Pradeep  Udhas). 


2.21 .1  Unique  or  Distinguishing  Practices 

The  Quality  Management  System  (QMS)  is  available  to  all  employees  through  the  Intranet. 
Past  data  from  projects,  best  practices  and  inputs  on  the  customer  are  shared  across  the  or¬ 
ganization  through  the  “process  database”  on  the  Intranet.  Organization  baselines  are  estab¬ 
lished  in  line  with  the  organization’s  quality  policy.  Each  project  sets  tailored  quality  objec¬ 
tives  based  on  the  organization  baselines  and  stage-wise  analysis  is  done  to  arrive  at 
corrective  actions  if  the  objectives  are  not  met.  Statistical  Process  Control  Methods  like  con¬ 
trol  charts,  Rayleigh  curves,  and  Gompertz  curves  are  used  widely  across  projects  to  measure 
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the  effectiveness  of  verification  and  validation  activities.  A  process  improvement  plan  is  de¬ 
veloped  and  this  forms  the  basis  for  our  process  improvement  activities.  The  SEPG  and  the 
SQA  group  are  responsible  for  process  improvement  initiatives  and  ensure  information  is 
disseminated  across  the  organization.  Our  future  goal  is  to  achieve  Level  5  in  year  2000. 

2.21.2  ROI  and  Improvement  Trend  Data 

A  key  factor  in  our  success  has  been  the  ability  to  show  that  our  efforts  have  had  value  to  the 
overall  organization.  As  a  result  of  widespread  use  of  data,  our  estimation  accuracy  has  im¬ 
proved  from  25%  to  12%,  our  outgoing  defect  density  has  reduced  from  3  defects/KLOC  to 
0.75  defects/KLOC,  and  there  has  been  a  25%  improvement  in  effort  overrun  from  what  it 
was  three  years  ago. 

2.21 .3  Issues  in  Achieving  High  Maturity 

We  still  have  a  long  way  to  go  to  achieve  our  ultimate  goal  of  continuous  process  improve¬ 
ment  as  a  way  of  life  in  the  organization.  Significant  issues  continue  to  place  challenges  in 
our  path.  These  include: 

•  Managing  growth  due  to  high  infusion  of  new  recruits. 

•  New  customers  with  unknown  process  maturity  levels. 

•  Knowledge  management. 

•  Better  alignment  of  business  goals  to  process  strategies  in  a  proactive  fashion. 

We  have  made  great  progress  over  the  last  three  years  and  we  expect  to  continue. 

2.21.4  ROI  and  Improvement  Trend  Data 

A  key  factor  in  our  success  has  been  the  ability  to  show  that  our  efforts  have  had  value  to  the 
overall  organization.  We  are  able  to  deliver  more  than  75%  of  our  projects  in  time  with  high 
degree  of  customer  satisfaction  resulting  from  very  low  outgoing  defects. 

2.21 .5  Issues  in  Achieving  High  Maturity 

We  still  have  a  long  way  to  go  to  achieve  our  ultimate  goal  of  continuous  process  improve¬ 
ment  as  a  way  of  life  in  the  organization.  Significant  issues  continue  to  place  challenges  in 
our  path.  These  include 

•  managing  growth  due  to  high  infusion  of  new  recruits 

•  new  customers  with  unknown  process  maturity  levels 

•  knowledge  management 

•  better  alignment  of  business  goals  to  process  strategies  in  a  proactive  fashion 
We  have  made  great  progress  over  the  last  three  years  and  we  expect  to  continue. 
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2.22  US  Army,  Communications  and  Electronics 

Command  (CECOM),  Software  Engineering  Center 
(SEC),  Fire  Support  Software  Engineering,  Fort 
Sill,  OK 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 


Point  of  Contact 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


4 

Nov  1997 
Don  Couch 
David  Zubrow 
Wolf  Goethart 

Don  Couch,  couchdc@fssec.army.mil 
Phil  Sperling,  sperlips@fssec.army.mil 
340 

20K-  1000K  SLOC 

Field  artillery,  message  processing,  communications,  operating  sys¬ 
tems 


2.23  US  Air  Force,  Directorate  of  Aircraft  Management, 
Software  Division,  Test  Software  and  Industrial 
Automation  Branches  (OC-ALC/LAS),  Tinker  AFB, 
OK 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


4 

Nov  1996 
Judah  Mogilensky 
Kelley  Butler 

Management  and  Technical  Support  Branch  Chief 

OC-ALC/LASM 

kelley .  butler  @  tinker .  af.mil 

350 

5K  to  100K  source  lines  of  code 

Avionics  test  software  development  and  maintenance 


The  Test  Software  and  Industrial  Automation  Branches  at  the  Oklahoma  City  Air  Logistics 
Center,  Tinker  AFB,  Oklahoma,  were  assessed  at  CMM  Level  4  in  November  1996.  This 
achievement,  the  first  Level  4  in  Federal  service,  was  significant  in  the  organization’s  quest 
for  continued  process  maturity  and  improvement.  We  have  furthered  strengthened  our  im- 
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provement  efforts  with  ISO  9001  Registration  in  November  1998.  These  efforts  were  further 
validated  in  May  1999  when  the  organization  was  awarded  the  IEEE  Award  for  Software 
Process  Achievement,  only  the  fifth  organization  to  receive  the  award  in  the  five  years  it  has 
been  in  existence.  This  discussion  will  address  the  software  organization,  its  demographics, 
the  organization’s  process  improvement  history,  return  on  investment,  and  the  issues  we  face 
as  we  strive  to  move  forward. 

The  Test  Software  and  Industrial  Automation  Branches’  primary  workload  is  avionics  test 
software  development  and  maintenance  (94%).  Other  functions  include  automatic  test  sys¬ 
tem  development,  jet  engine  testing,  jet  engine  trending,  non-destructive  inspection,  and  sup¬ 
port  of  the  jet  engine  overhaul  process.  There  are  approximately  350  software  professionals. 
This  compares  to  just  90  in  1990,  an  almost  400  percent  increase  in  the  past  nine  years.  Our 
process  improvement  successes  along  with  our  improvements  in  productivity  and  defects 
have  directly  resulted  in  our  growth.  Our  team  sizes  range  from  3  to  60  with  test  program 
sizes  from  5K  to  100K  source  lines  of  code.  It  should  be  noted  that  our  projects  consist  of 
multiple  test  programs,  ranging  from  one  to  more  than  one  hundred. 

The  Test  Software  and  Industrial  Automation  Branches  have  focused  on  Software  CMM- 
based  improvement  since  1990  when  our  first  SEI-based  assessment  was  performed.  In 
March  1993,  one  month  after  the  release  of  Software  CMM  v.1.1,  the  first  CMM  based  as¬ 
sessment  was  performed  by  the  SEI  at  our  site,  resulting  in  a  Level  2  rating.  Judah  Mogilen- 
sky  led  our  1996  Level  4  assessment  where  two  of  the  three  Level  5  KPAs  were  also  deemed 
satisfied.  The  only  KPA  we  did  not  satisfy,  Defect  Prevention,  has  been  strengthened  by  our 
implementation  of  ISO  9001,  including  the  Corrective  and  Preventive  Action  clause.  We 
were  registered  to  ISO  9001/TickIT  in  November  1998. 

A  key  factor  in  our  success  has  consistent  and  active  leadership.  With  the  exception  of  a 
leadership  change  in  one  branch  and  the  addition  of  two  branches  due  to  growth,  our  senior 
organizational  leadership  has  not  changed  since  1990.  We  have  also  maintained  a  consistent 
focus  on  our  customers  and  on  implementing  improvements  that  truly  benefit  the  organiza¬ 
tion. 


2.23.1  ROI  and  Improvement  Trend  Data 

We  continue  to  refine  and  examine  our  return  on  investment  to  make  sure  that  we  are  truly 
making  the  progress  we  expect.  Additionally,  “stretch”  productivity  goals  are  set  each  year 
for  our  branches.  Our  latest  data  shows  our  software  quality  (reduction  of  defects)  and  pro¬ 
ductivity  (man-hours  and  months  per  Test  Program  Set). 


Defects  per  KSLOC 

•  Average  3.3  in  early  1990s 

•  Average  0.4  in  mid  1990s 
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Latest  Data  showing  0.06  defects  per  KSLOC  (1998  TPSs) 


TPS  Development  Productivity 

•  1600  Mhrs/13  Months  per  TPS  -  1993 

•  1200  Mhrs/12  Months  per  TPS  -  1996 

•  1150  Mhrs/12  Months  per  TPS  - 1997 

•  923  Mhrs/1 2  Months  per  TPS  - 1 998 

•  1081  Mhrs/1 8  Months  per  TPS  - 1999 

*B-2  Program,  Contractual  Obligations  increased  man-hours  and  decreased  cycle  time 

2.23.2  Issues  in  Achieving  High  Maturity 

We  feel  that,  while  our  improvement  efforts  have  been  very  successful,  we  now  have  to  place 
additional  emphasis  on  sustaining  and  continuing  our  efforts — continuous  and  incremental 
improvement.  This  is  an  area  that  is  directly  benefited  by  our  ISO  9001  efforts  and  the  re¬ 
quired  third-party  audits.  One  area  that  we  feel  that  we  excel  in  is  a  highly  disciplined 
maintenance  process  which  closely  mirrors  our  development  process  and  gives  our  customers 
confidence  in  the  changes  that  we  are  making  to  their  mission  critical  software. 

Our  biggest  Maturity  Level  4/5  issue  is  determining  the  proper  application  of  SPC  for  Soft¬ 
ware.  Recent  focus  has  been  on  the  application  of  SPC  for  Project  Management  as  well  as 
the  proper  use  of  cost  and  schedule  reserve.  Additional  focus  has  also  been  placed  on  proper 
accounting  of  requirement  changes  and  re-planning  efforts.  We  are  working  to  use  SPC  in  a 
manner  that  benefits  the  organization  and  adds  value  to  our  process. 

2.24  Wipro  GE  Medical  Systems,  Bangalore,  India 

5 

Jan  1999 
Richard  Knudson 
Rama  Rao 
K.V.  Ramaswami 
Six  Sigma  Master  Black  Belt 
kv.ramaswami@geind.ge.com 
http://www.wipro-ge.com/ 

Software:  170,  Total:  650 
Medical  scanners 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 

Point  of  Contact 

Web  Page 

Size  of  the  Organization 
Primary  Application  Domain 
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2.24.1  ROI  and  Improvement  Trend  Data 


1997 

1998 

1999 

Effort  Overruns  (Actual/Estimated) 

Mean 

135% 

105% 

98% 

Std.  Deviation 

56% 

12% 

n% 

%Projects  with  >  10%  overrun 

70% 

5% 

0% 

Schedule  Overruns  (Actual/Estimated) 

Mean 

142% 

113% 

100% 

Std.  Deviation 

84% 

30% 

7% 

%Projects  with  >  10%  overrun 

70% 

30% 

0% 

Quality  in  Sigma  Terms  (from  Defects  /  Million 
Test  Cases) 

3.3 

4.5 

5.2 

Rework  Effort  %  of  total  project 

12% 

3% 

1% 

2.24.2  Unique  or  Distinguishing  Practices 

Six  Sigma  for  process  and  product  quality  (through  design)  improvement 
One-to-one  mapping  of  quality  models  to  business  models 

2.24.3  Issues  in  Achieving  High  Maturity 

Creating  one  look  and  feel  for  user 
Integrating  ISO  and  EN46001 
FDA/QSR  2.0  and  Six  Sigma  methods 


2.25  Wipro  Technologies,  Global  R  &  D  Division, 
Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 

Point  of  Contact 


5 

June  1999 
Richard  Knudson 
Mark  Paulk 
V.  Subramanyam 

Technical  Manager  (Process  and  Quality) 
Subramanyam.venkat@wipro.com 
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Web  Page  www.wipro.com 

Size  of  the  Organization  Over  2000  software  professionals 

Typical  Program  Size  Small  and  medium  sized  (10-100  KSLOC)  with  maximum  up  to  500 

KSLOC 

Primary  Application  Domain  System  software,  communications  and  networking  (datacom  and  tele¬ 
com)  and  embedded  systems  and  IP  enablers.  Design  solutions  for 
VLSI/ASIC  ,  HAV  circuits  as  well  as  system  integration  support 


WIPRO  Technologies,  Global  R  &  D  Division  is  a  part  of  WIPRO  Corporation,  the  largest 
publicly  traded  company  in  India. 

Global  R  &  D  Division  offers  design  service  to  the  large  global  platform  and  equipment  ven¬ 
dors  for  the  Internet  infrastructure.  This  is  done  by  deriving  expertise  from  over  2000  engi¬ 
neers  in  the  areas  of  Platform  Technologies,  Communication  and  Networking  Technologies, 
Embedded  Systems  Technologies  and  VLSI  /  System  Design. 

Global  R  &  D’s  Quality  journey  started  in  1992  with  a  formal  ISO  9001  initiative.  The  divi¬ 
sion  was  certified  for  ISO  9001  in  Sept  1994.  Soon  after  that,  the  division  continued  further 
process  improvement  initiatives  based  on  the  SEI CMM  model.  Global  R  &D  was  assessed  at 
SEI  CMM  Level  4  on  March  1997  and  at  SEI  CMM  Level  5  in  June  1999.  Focus  during  the 
Quality  journey  is  to  plan  and  implement  process  improvement  activities  based  on  business 
need,  to  meet  customer  requirements  and  to  overcome  the  current  pain  areas  facing  the  proj¬ 
ect  teams. 

2.25.1  ROI  and  Improvement  Trend  Data 

Some  of  the  benefits  realized  from  the  process  improvement  program  are 

•  continuous  improvement  trend  in  customer  satisfaction  level  as  re¬ 
flected  by  periodic  customer  satisfaction  survey  results. 

•  control  of  project  schedule  and  effort  overruns.  Less  than  10%  of 
projects  had  schedule  overrun  and  less  than  15%  of  projects  had  effort 
overrun  during  the  year  98-99. 

•  reduced  detection  of  acceptance  defect  over  the  years  after  the  release 
of  work  products  to  the  customer 

•  high  level  of  in-process  defect  removal  efficiency  of  over  95  %  from 
past  three  years  and  98  %  during  the  year  98-99 

•  formal  reviews  institutionalized  and  considered  as  strength  of  the  or¬ 
ganization  and  helping  the  project  teams  in  catching  around  85%  of 
defects  within  the  phase 

•  widening  the  scope  of  business  in  different  geographies  and  in  new 
areas  of  domains 
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2.25.2  Unique  or  Distinguishing  Practices 

Some  of  the  best  practices  institutionalized  during  the  quality  journey  are 

•  a  single  integrated  documented  web  based  quality  system  and  process 
assets  accessible  to  all  employees  on  the  intranet 

•  systematic  collection  and  analysis  of  metrics  data  at  project  level  and 
well  defined  DP  activities  like  Look  Ahead  Meetings  and  root  cause 
analysis 

•  implementation  of  role  called  Quality  Improvement  facilitators  for 
process  facilitation  and  process  consultancy  to  project  teams 

•  project  Managers  owning  the  responsibility  of  project  level  QA  ac¬ 
tivities 

•  rigorous  audit  mechanism  with  focus  areas  defined  for  each  audit. 
Supplemented  by  periodic  CMM  based  process  assessments 

•  use  of  Query  based  tool  called  HPD  (Historical  Project  Database)  to 
query  past  projects  data  on  various  parameters  for  better  estimation 
and  for  planning  defect  prevention  activities 

•  dedicated  tools  group  working  on  development  and  implementation  of 
process  automation  tools  to  automate  software  engineering  practices 
of  the  division 

•  use  of  web  based  in-house  tools  for  capturing  effort  ETS  (  Effort 
Tracking  system),  Review  defects  ARTS  (Automatic  Review  tracking 
system)  and  DeLTA  (Defect  Logging  and  tracking  system)  etc., 

•  use  of  Six  Sigma  methodology  for  improving  selected  business  criti¬ 
cal  transactional  processes 


2.25.3  Issues  in  Achieving  High  Maturity 

Working  with  customers  of  different  process  maturity 
Need  to  continuously  train  practitioners  due  to  high  growth  rate 
Non  repetitive  projects  in  diverse  business  domains 
Deriving  benefits  of  metrics  analysis  for  short  duration  projects 
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2.26  Wipro  Technologies,  Enterprise  Solutions 
Division  (ESD),  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor 
Point  of  Contact 


Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain 


5 

December  1998 
Richard  S  torch 
Subbarao  TV 
GM-QA 

subbarao.tangirala@wipro.com 

www.wipro.com 

4000  software  professionals 

medium  sized  (50  -  75  KSLOC)  with  maximum  up  to  1000  KSLOC 

e-commerce,  Enterprise  application,  data  warehousing,  infrastructure 
management,  client  server,  legacy  maintenance,  system  software,  em¬ 
bedded  systems,  telecom 


Wipro  Technologies,  ESD  is  a  part  of  WIPRO,  a  global  provider  for  software  and  services  to 
Fortune  500  companies.  It  is  the  largest  publicly  traded  company  of  India. 

Wipro  ESD  was  assessed  at  SEI CMM  Level  5  on  Dec  1998,  based  on  CBA IPI  method.  The 
Lead  Assessor  was  Richard  F.  Storch.  A  well  defined  framework  for  analysis,  approval  and  im¬ 
plementation  of  process  improvement  proposals  from  all  employees,  ISO  audits,  internal  as¬ 
sessments,  continuous  improvement  councils,  top  management  review  and  Six  Sigma  teams 
form  the  cornerstone  for  continuous  improvement 

4000-plus  highly  skilled  software  professionals  form  the  backbone  of  the  organization.  Main 
service  offerings  are  in  e-commerce,  Enterprise  application,  data  warehousing,  infrastructure 
management,  client  server,  legacy  maintenance,  system  software,  embedded  systems  and  tele¬ 
com.  The  industry  segment  focus  is  in  finance,  retail,  utility,  healthcare,  corporate,  and  telecom. 
The  typical  program  size  are  medium  sized  (50  -  75  KSLOC)  with  maximum  up  to  1000 
KSLOC.  Total  number  of  projects  being  executed  at  geographically  distributed  development 
center  is  approximately  200. 

The  foundation  for  quality  improvement  journey  was  established  in  1992  by  an  ISO  9000  ini¬ 
tiative  to  formulate  disciplined  processes  for  conducting  software  operations.  Having  achieved 
ISO  9001  Tick  IT  standard  in  1995,  we  selected  the  SEI  CMM  model  to  improve  process  ma¬ 
turity  further.  The  first  milestone  was  reaching  SEI  CMM  Level  3  in  Jun  1997.  Process  im¬ 
provement  journey  continued  and  we  reached  SEI  CMM  Level  5  maturity  in  Dec  1998.  Process 
improvement  continues  by  all  employees  taking  part  in  the  journey,  with  additional  focus  by 
leveraging  on  Six  Sigma  methodologies.  Six  Sigma  approach  focuses  on  defect  reduction  and 
cycle  time  reduction  by  enforcing  data  centered  approach,  customer  focus  and  measuring  finan¬ 
cial  benefit  from  process  improvement  effort.  The  Six  Sigma  milestone  is  to  reach  4-sigma  by 
March  2000  and  6  sigma  by  Dec  2002  in  all  the  critical  business  processes. 
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2.26.1 


ROI  and  Improvement  Trend  Data 


Some  of  the  benefits  realized  from  the  process  improvement  program  are 

•  measured  and  predictive  processes  with  clearly  defined  metrics  for  Effort  overrun. 
Schedule  overrun,  Rejection  index,  productivity  indices,  Customer  satisfaction  index  etc. 

•  well  defined  defect  prevention  at  project  level  involving  defect  reporting,  root  cause 
analysis,  preventive  action  plan,  implementation  and  prevention  feedback 

•  control  of  project  schedule  overruns.  Almost  70  %  of  56  projects  in  98-99  were  com¬ 
pleted  within  or  on  time 

•  improved  estimation  capability  due  to  increased  use  of  metrics.  There  has  been  marked 
decline  in  effort  overruns.  Statistics  shows  a  decline  from  20  %  in  1996  to  6  %  in  1998. 

•  reduced  defects  and  minimization  of  rejections  due  to  increase  process  focus  at  every 
stage  has  resulted  in  increased  productivity  .  It  has  been  observed  that  around  79  %  of  the 
92  projects  completed  in  1998  had  a  defect  ratio  of  below  1  error/  KSLOC 

•  reduced  risk  due  to  institutionalization  of  formal  risk  management  procedures  for  risk 
identification,  assessment  and  control. 


2.26.2  Unique  or  Distinguishing  Practices 

Some  of  the  practices  which  is  helpful  in  our  quality  journey  are 

•  web  based  quality  system  accessible  by  all  employees  across  geographical  locations  with 
additional  features  for  initiation  of  process  improvement  proposals,  review  by  domain 
experts  and  consolidation  by  SEPG 

•  web  based  tool  for  the  conduct  of  company  wide  ISO  audits 

•  automation  tools  for  project  status  reporting  at  different  levels 

•  tools  for  resource  (hardware,  software  and  human)  tracking  and  allocation,  performance 
appraisal,  training  management  etc 

•  project  based  Six  Sigma  approach  for  defect  reduction  and  cycle  time  reduction. 

Other  initiatives  include  People  CMM. 

2.26.3  Issues  in  Achieving  High  Maturity 

Identifying  measurements  for  diverse  business  domains. 

Application  of  control  charts  in  small  projects 

Automation  for  metrics  collection 

Convincing  customer  about  the  need  to  improve  process  maturity  to  higher  levels. 
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3  Preliminary  Results  of  the  1999  High 
Maturity  Survey 


The  preliminary  results  of  a  survey  of  high  maturity  practices  were  presented  to  the  work¬ 
shop.  The  final  results  of  the  survey  have  been  published  as  an  SEI  special  report  [Paulk  00] 
and  represent  inputs  from  37  Level  4  or  5  organizations.  The  preliminary  results  were  based 
on  responses  from  32  high  maturity  organizations  from  the  40  Level  4  and  21  Level  5  organi¬ 
zations  known  to  have  been  assessed  as  of  November  1999.  This  includes  26  non-US  high 
maturity  organizations:  24  organizations  in  India,  with  14  at  Level  4  and  10  at  Level  5;  one 
Level  4  organization  in  Australia,  and  one  Level  4  organization  in  Israel. 

Of  the  26  high  maturity  organizations  that  participated  in  the  workshop,  12  were  from  India, 
suggesting  that  the  participation  was  reasonably  representative  of  the  worldwide  high  matur¬ 
ity  community. 

For  the  survey  results,  see  <URL:  http://www.sei.cmu.edU/cmm/cmm.articles.html#hmp99>. 
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4  Working  Group  Reports 


Working  groups  were  identified  in  eight  areas: 

•  statistics 

•  assessments 

•  CMM  integration 

•  measurement 

•  technology  transition 

•  human  issues 

•  reorganizations 

•  software  quality  assurance 

There  was  sufficient  interest  in  the  working  groups  on  statistics  and  measurement  to  split  them  into  two 
different  sessions;  each  is  independently  reported  on  in  this  section.  No  one  signed  up  for  the  working 
group  on  reorganizations,  so  this  working  group  was  cancelled. 


4.1  Working  Group  1 :  Statistics  (Session  A) 

Participants:  Mark  Amaya,  Colin  Benton,  Bill  Curtis  (presenter),  Rick  Hefner,  Pankaj  Jalote,  Keith  Joyce, 
Barbara  Kolkhorst,  Jane  Moon,  Andre  Heijstek,  Steve  Janiszewski,  Mark  Paulk  (facilitator),  Neil  Potter, 
Leitha  Purcell,  Phil  Sperling,  Ramesh  Venkatraman,  Ramaswami  K.  Viswanathan  (scribe),  Don  White 
(timekeeper),  and  Gary  Wigle. 

At  the  beginning  of  the  working  group  discussion,  a  number  of  books  on  statistical  process  control  (SPQ 
were  recommended  by  Mark  Paulk: 

•  Measuring  The  Software  Process  by  William  A.  Horae  and  Anita  D.  Carleton,  Addison  Wesley 

•  Building  Continual  Improvement:  A  Guide  for  Business  by  Donald  J.  Wheeler  and  Sheila  R.  Poling, 
SPC  Press 

•  Understanding  Statistical  Process  Control  by  Donald  J.  Wheeler  and  David  S.  Chambers,  SPC  Press 

•  Understanding  Variation:  The  Key  to  Managing  Chaos  by  Donald  J.  Wheeler,  SPC  Press 

•  Advanced  Topics  in  Statistical  Process  Control:  The  Power  of  Shewhart’s  Charts  by  Donald  J. 
Wheeler,  SPC  Press 

•  Short  Run  SPC  by  Donald  J  Wheeler,  SPC  Press 
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A  handout  was  also  provided  to  expand  on  an  earlier  discussion  on  SPC  terminology  in  the  plenary  session. 
In  a  column  entitled  “AModest  Proposal,”  Don  Wheeler  has  recommended  some  changes  in  the  traditional 
SPC  terminology  that  may  help  address  some  of  the  misunderstandings  of  SPC,  which  is  summarized  in 
Table  1. 


Table  1:  Wheeler’s  Proposed  New  SPC  Terminology 


Traditional  SPC  Term 

Wheeler’s  Preferred  New  Term 

;  \\ 

Statistical  Process  Control 

Continual  Improvement 

Control  Charts 

Process  Behavior  Charts 

In-Control  Process 

Predictable  Process 

Out-of-Control  Process 

Unpredictable  Process 

Out-of-Control  Point 

Point  Outside  the  Limits 

In-Control  Point 

Point  Inside  the  Limits 

Control  Limits  for  Individual  Values 

Natural  Process  Limits 

Control  Limits  for  Averages 

Limits  for  Averages 

Control  Limits  for  Ranges 

Limits  for  Ranges 

Paulk  proposed  a  number  of  hypotheses  regarding  quantitative  management  as  a  starting 
point  for  the  working  group  to  discuss,  which  led  to  observations  and  recommendations. 

Five  topics  were  voted  of  most  interest  to  the  participants.  They  proposed  that  a  true  maturity 
Level  4  organization  can  and  will 

1 .  show  significant  improvement  trends  in  business  results  relative  to  process  improvement 

2.  demonstrate  capable  processes  at  Level  4 

3.  control  processes  at  the  activity  /  step  level  of  granularity  on  a  day-to-day  basis 

4.  demonstrate  the  application  of  quantitative  management  using  “rigorous”  techniques  to 
control  the  design  (architectural  and  detailed),  coding,  and  testing  processes  (and  proba¬ 
bly  requirements  analysis) 

5.  identify  causes  of  stratification  in  process  data 

4.1.1  Observations 

The  working  group  members  were  surprised  that  there  are  so  many  different  opinions  about 
what  constitutes  SPC.  It  was  clear  that  there  is  little  common  understanding  in  the  SEI  and 
software  process  improvement  (SPI)  communities  of  what  SPC  and  Quantitative  Process 
Management  (QPM)  are  or  imply.  SPC  is  new  to  the  software  community;  it  is  easy  to  mis¬ 
apply,  reach  wrong  conclusions,  and  make  poor  decisions. 

One  example  of  this  is  the  use  of  sample  standard  deviation  as  an  estimator  for  sigma  when 
setting  control  limits.  Any  introductory  statistics  course  teaches  that  the  standard  deviation  of 
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the  sample  approximates  the  value  of  sigma  for  the  population.  Since  spreadsheet  programs 
usually  include  a  “stdev”  function,  it  is  easy  to  calculate  3-sigma  control  limits  using  “stdev.” 
Unfortunately,  there  are  several  ways  to  estimate  sigma,  and  the  sample  standard  deviation 
happens  to  be  a  bad  one  for  control  charts  because  it  combines  common  causes  of  variation 
(desirable)  with  assignable  causes  of  variation  (undesirable).  This  inflates  the  control  lim¬ 
its — sometimes  by  factors  as  great  as  3-5  times  larger  than  they  would  be  using  the  correct 
formulas  (Don  Wheeler,  "Charts  Done  Right,"  Quality  Progress,  June  1994;  Thomas  Pyzdek, 
"How  Do  I  Compute  a?  Let  Me  Count  the  Ways,"  Quality  Digest,  May  1998). 

Similarly,  many  of  the  problems  with  large  variation  in  the  software  process  may  result  from 
aggregating  multiple  processes,  with  a  resultingly  large  set  of  control  limits.  One  example 
reported  at  the  workshop  was  for  a  control  chart  in  which  the  3-sigma  limits  were  considered 
too  wide.  A  ruler  was  moved  down  the  chart  until,  at  about  1.5  sigma,  “signals”  were  de¬ 
tected  in  the  data  based  on  a  judgment  that  the  testing  error  rates  had  been  problematic.  In 
this  case,  the  main  modules  used  to  control  the  program  had  been  considered  simple,  and  an 
“abbreviated”  process  was  used  to  build  the  main  control  modules.  In  testing,  however,  these 
simple  modules  turned  out  not  to  have  been  that  easy  to  program.  It  was  pointed  out  that  this 
is  actually  an  example  of  stratification — combining  data  from  separate  processes — in  this 
case,  the  normal  software  process  and  an  abbreviated  process.  There  may,  or  may  not,  have 
been  signals  in  the  data  for  each  of  these  processes,  but  it  seemed  clear  that  the  abbreviated 
process  was  not  capable  of  providing  the  quality  needed  for  the  main  control  modules. 

The  issue  of  large  amounts  of  variation  in  the  software  process  remains  a  significant  one  re¬ 
gardless  of  disaggregation  techniques,  such  as  identifying  stratification.  Individual  differ¬ 
ences  are  massive  in  intellectual  activities  such  as  software  engineering.  While  teamwork, 
disciplined  processes,  and  disaggregation  minimize  this  intrinsic  variation,  no  consensus  was 
reached  on  whether  organizations  should  expect  to  be  able  to  apply  control  charts  to  the  soft¬ 
ware  process  in  all  circumstances,  although  many  examples  of  their  application  were  men¬ 
tioned. 

The  working  group  members  observed  that  it  is  hard  to  establish  the  relationship  between 
SPC  and  business  objectives  at  the  executive  level.  Quantification  of  benefits  is  not  a  com¬ 
mon  practice  in  most  organizations  regardless  of  maturity,  but  it  was  also  observed  that  a  true 
Level  4  organization  can  identify  the  business  objectives  it  wants  to  control  through  quantita¬ 
tive  management  practices.  It  is  easier  to  identify  the  relationships  to  intermediate  business 
objectives,  such  as  decreased  cycle  time  and  defect  densities,  than  to  executive-level  business 
objectives  such  as  increased  market  share  and  stock  price.  Many  executive-level  business 
objectives  may  be  viewed  as  having  only  a  tenuous  connection  to  process  improvement  or 
quality. 

The  working  group  members  observed  that  they  only  heard  people  applying  quantitative 
management  to  a  few  processes,  such  as  inspections  and  earned  value  analysis.  There  was  a 
discussion  on  whether  controlling  the  inspection  of  the  design,  for  example,  should  be  con- 
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sidered  as  controlling  the  design  process  as  well  as  the  inspection  process,  since  causal  analy¬ 
sis  may  lead  to  problems  in  the  design  process  as  well  as  the  inspection  process.  There  was 
general  agreement  within  the  working  group  that  few  examples  of  control  charts  in  the  early 
phases  of  the  life  cycle,  especially  requirements  analysis,  were  available.  It  was  also  noted 
that  the  high  maturity  practices  survey  indicates  that  control  chart  usage  in  the  requirements 
phase  is  comparable  to  that  in  later  life  cycle  phases. 

4.1 .2  Recommendations  for  High  Maturity  Organizations 

The  working  group  identified  a  number  of  recommendations  for  high  maturity  organizations 
to  consider. 

First,  Level  4  organizations  should  provide  the  project  teams  with  sufficient  training,  tools, 
and  mentoring  in  statistical  and/or  modeling  methods  to  apply  appropriate  quantitative  man¬ 
agement  techniques  effectively. 

Second,  data  collection  should  be  frequent  enough  to  provide  real-time  control  of  the  process. 
Whether  this  control  should  be  on  a  daily  or  weekly  basis  was  discussed,  but  no  consensus 
was  reached  since  circumstances  may  vary. 

Third,  understanding  variation  is  required  at  Level  4,  but  not  the  use  of  control  charts.  A  high 
maturity  organization  should  choose  the  appropriate  statistical  or  modeling  technique  to  an¬ 
swer  the  specific  questions  that  it  has.  Control  charts  are  a  powerful  statistical  tool,  but  many 
other  rigorous  techniques  are  possible,  such  as  analysis  of  variance  and  regression  analysis. 
One  example  of  a  non-SPC  quantitative  management  technique  that  was  mentioned  is  the 
weighted  lateness  technique  developed  by  Telcordia  Technologies. 

Fourth,  a  Level  4  organization  is  not  required  to  have  capable  processes,  but  it  must  under¬ 
stand  the  capability  of  its  processes.  Process  capability  implies  a  statistical  understanding  of 
the  expected  performance  of  a  process  in  terms  of  expect  mean  and  variation.  Process  capa¬ 
bility  indices  are  frequently  used  to  compare  expected  performance  based  on  3-sigma  limits 
versus  the  requirements  (specification  limits)  for  the  process.  A  capable  process  is  one  that 
can  satisfy  its  requirements  in  its  normal  operation,  i.e.,  its  3-sigma  limits  are  within  the 
specification  limits.  Even  in  a  Level  4  or  5  organization,  the  requirements  set  for  a  process  by 
the  customer,  whether  internal  or  external,  may  not  be  achievable  within  current  process  ca¬ 
pability. 

There  was  an  extended  discussion  about  what  a  Level  4  or  5  organization  can  and  will  do 
about  this.  There  are  no  guarantees  of  success,  even  in  high  maturity  organizations,  of 
adopting  new  processes  that  will  be  capable.  In  Level  4  organizations,  disparities  in  process 
capability  and  requirements  will  be  recognized,  and  the  project  may  take  corrective  action — 
including  changing  the  requirements— to  address  the  problem.  In  Level  5  organizations, 
there  is  an  ongoing  systematic  effort  to  improve  process  capability  across  projects.  These 

organizational  efforts  may  help  projects  address  process  capability  prohlems,  hut  high - 
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izational  efforts  may  help  projects  address  process  capability  problems,  but  high  maturity 
promises  an  understanding  of  the  problem  rather  than  a  successful  resolution. 

Fifth,  continue  disaggregating  process  control  data  until  the  chart  is  usable.  Then  people  can 
take  action  on  the  data  and  be  held  accountable.  Aggregated  data  is  too  variable  by  defini¬ 
tion. 

Sixth,  the  use  of  SPC  in  earlier  phases  of  the  life  cycle,  such  as  requirements  analysis,  should 
be  institutionalized. 

4.1.3  Recommendations  for  the  SEI 

The  working  group  identified  a  number  of  recommendations  for  the  SEI  to  consider. 

First,  the  SEI  should  clarify  to  the  SPI  community  whether  the  current  operating  model  is 

•  Software  CMM  Version  1.1  as  written 

•  Version  1.1  as  reinterpreted,  clarified,  and  elaborated  in  Software  CMM  Version  2  Draft  C 

•  what  we  wish  we  had  said  when  we  wrote  Version  1.1,  which  is  not  exactly  what  we  said 

The  working  group  recommended  the  first  option,  Version  1.1  as  written,  at  least  until  a  re¬ 
placement  model,  presumably  based  on  the  ongoing  CMM  Integration  work,  is  provided. 
Although  the  SEI  and  the  SPI  community  have  learned  much  about  quantitative  management 
and  SPC  since  Version  1.1  was  released,  for  reliability  and  consistency,  the  operating  model 
must  be  Version  1.1.  There  are  concerns  about  inconsistency  of  interpretations,  primarily  at 
Level  4,  and  these  need  to  be  addressed,  but  it  should  be  in  the  context  of  Version  1 . 1  as 
written. 

Second,  the  SEI  should  clarify  confusing  high  maturity  issues  in  CMMI  model  for  Level  4  at 
the  goal  and  practice  level.  It  was  noted  that  the  review  period  for  CMMI  v0.2,  currently  un¬ 
der  review,  ends  November  30,1999,  and  the  workshop  attendees  were  encouraged  to  partici¬ 
pate  in  the  review,  particularly  from  a  high  maturity  perspective. 

Third,  the  SEI  should  maintain  flexibility  in  the  range  of  quantitative  methods  that  are  legiti¬ 
mate  (or  required)  at  Level  4.  Many  different  quantitative  methods  can  be  used  to  support 
quantitative  management. 

Fourth,  the  SEI  should  get  input  from  more  organizations  in  building  consensus  on  high  ma¬ 
turity  practices,  and  disseminate  this  information  for  review  and  guidance  to  organizations 
that  need  its  guidance. 

Fifth,  the  SEI  should  publish  a  compendium  of  quantitative  management  practices  (including 
examples  other  than  SPC)  currently  in  use  and  their  benefits. 
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Sixth,  the  SEI  should  request  Lead  Assessors  to  supply  case  studies  on  Level  4  and  5  organi¬ 
zations. 

Seventh,  the  SEI  should  create  guidelines  for  applying  quantitative  management  techniques 
based  on  industry  lessons  learned. 

Eighth,  the  SEI  should  not,  however,  be  the  final  authority  on  statistics.  This  last  point  was 
inspired  by  a  discussion  of  2-sigma  versus  3-sigma  control  limits.  Some  high  maturity  or¬ 
ganizations  are  using  2-sigma  limits  to  trigger  action,  and  it  was  observed  that  several  SEI 
statistics  experts  have  commented  that  these  are  not  valid  control  limits.  Paulk  stated  that,  in 
his  judgment,  “action  limits”  based  on  2-sigma  were  legitimately  based  on  an  understanding 
of  variation  and  thus  could  be  considered  a  valid  quantitative  management  technique  at  Level 
4.  Some  statisticians  consider  these  to  be  valid  control  limits;  other  statisticians  note  that  this 
is  an  explicit  violation  of  Shewhart's  rationale  for  choosing  3-sigma  limits  and  state  that  2- 
sigma  limits  are  incorrect.  Given  the  rift  in  the  statistical  community  on  this  issue,  anyone 
using  2-sigma  limits  should  be  educated  that  this  is  not  generally  accepted  practice,  and  the 
use  of  the  term  “2-sigma  control  limits,”  unless  made  in  ignorance,  is  likely  to  be  considered 
as  taking  a  position  in  a  heated  debate  that  makes  the  proponent  fair  game.  It  is  fair  to  say 
that  most  SEI  staff  knowledgeable  in  SPC  are  aligned  with  Don  Wheeler’s  philosophies,  so 
characterizing  2-sigma  limits  as  control  limits  is  likely  to  lead  to  a  correction  in  terminology. 

4.2  Working  Group  1 :  Statistics  (Session  B) 

Participants:  Bijay  Kumar  Jyotishi,  Anand  Kumar,  Walt  Lipke,  Andy  Meadow,  Mary  Lynn 
Penn,  Joseph  Seppy,  Rakesh  Soni,  Subramanyam  V.,  and  Charlie  Weber. 

4.2.1  What  is  Level  4  Really  Trying  to  Accomplish? 

Level  4  is  an  evolving  interpretation  using  data,  leading  to  statistical  process  control.  The 
group  discussed  at  length  whether  Level  4  is  limited  to  ensuring  the  stability  of  the  software 
process,  or  it  also  requires  process  improvement.  Consensus  on  this  issue  proved  elusive,  al¬ 
though  the  vast  majority  of  the  members  of  the  group  believe  that  Level  4  is  limited  to  stabil¬ 
ity  of  the  software  process. 

One  opinion  raised  during  the  discussion  was  that  Level  4  is  actually  a  transient  level,  be¬ 
tween  Levels  3  and  5.  Organizations  cannot  really  sustain  Level  4,  without  moving  towards 
Level  5.  If  they  choose  not  to,  they  will  probably  slide  back  to  Level  3.  Obviously,  this  issue 
could  not  be  discussed  threadbare  due  to  constraints  of  time. 

The  group  agreed  on  the  fact  that  Level  4  has  a  product/process/project  orientation  for  proc¬ 
ess  improvement,  whereas  Level  5  implies  organization-oriented  improvement.  At  Level  4, 
you  need  to  reconcile  the  project’s  process  capability  with  performance  goals/commitments. 
In  order  to  achieve  this,  the  process  must  be  planned  and  defined,  along  with  attainable  qual- 
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ity  goals  that  are  consistent  with  one  another.  A  related  view  expressed  by  some  members  of 
the  group  was  that  the  planned  processes  must  be  based  on  past  process  performance,  plus 
defined  strategies,  to  address  the  estimated  gap  between  past  performance  and  the  quality 
goals  of  the  project.  However,  there  was  no  consensus  on  this  last  point. 

4.2.2  How  Can  You  Better  Prepare  for  Level  4? 

In  order  to  be  geared  for  Level  4,  organizations  need  to  look  ahead  when  implementing  lower 
levels  of  the  CMM.  For  example,  the  automation  of  data  collection  should  be  in  place  before 
getting  to  Level  4,  so  that  the  quantitative  focus  of  Level  4  does  not  add  overheads  to  the  im¬ 
plementation  of  the  software  process.  This  look-ahead  requires  that  members  of  the  organi¬ 
zation,  who  are  charged  with  leading  the  software  process  improvement  program  at  Levels  2 
and  3,  are  trained  on  the  higher  level  KPAs  as  well.  For  example,  they  should  be  able  to  un¬ 
derstand  that  run  charts  could  lead  to  control  charts.  The  group  felt  that  the  M&A  KPA  at 
Level  2  of  the  CMMI  could  facilitate  this  transition. 

4.2.3  Statistical  Techniques 

What  do  we  mean  by  statistical  methods? 

What  new  statistical  techniques  can  we  use  when  moving  from  Level  4  to  5? 

Is  there  some  way  of  classifying  these  techniques? 

Do  we  have  examples  of  where  statistics  have  /  have  not  worked? 

After  intense  discussions,  the  group  was  able  to  reach  consensus  on  the  following  classifica¬ 
tion  of  statistical  techniques: 

•  “Startup”  Techniques 

-  Computation  of  mean  /  mode  /  median 

-  Run  charts 

•  “Startup  /  Intermediate”  Techniques 

-  Histograms 

-  Pareto  charts 

•  “Intermediate”  Techniques 

-  Control  charts 

-  Correlation  charts 

-  Regression  analysis 

-  Fishbone  diagrams 

-  Rayleigh  curves 

-  Orthogonal  defect  classification 

•  “Advanced”  Techniques 

-  Process  models  /  simulation 
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-  Analysis  of  variance  /  mean 

•  “Not  Sure  /  Advanced”  Techniques 

-  Multivariate  analysis 

-  Design  of  experiments 

-  Prediction  intervals 

Members  of  the  group  were  able  to  recount  some  of  their  experiences  on  statistical  tech¬ 
niques  that  did  or  did  not  work  in  their  organizations. 

The  following  techniques  appear  to  have  worked: 

•  run  charts  for  productivity,  rework,  and  program  measures 

•  control  charts  for  program  measures  and  quality  (most  organizations  seem  to  use  XmR 
for  this) 

•  Rayleigh  curves  for  defect  prevention 

•  orthogonal  defect  classification  for  defect  prevention,  causal  analysis,  process  change 
management,  and  technology  change  management 

•  correlation  charts  /  scatter  diagrams  for  the  relationship  between  requirements  volatility 
and  product  quality,  and  for  predicting  size  against  effort  and  defects 

•  regression  analysis  to  estimate  effort  and  schedule  from  size 

•  histogram  for  effort  and  schedule  overrun 

The  following  techniques  appear  to  have  not  worked: 

•  run  charts  used  on  test  defect  data 

•  combining  data  from  various  domains  and  constructing  histograms 

•  u  charts  for  data  that  has  large  intra-sample  differences 

•  usage  of  control  charts  in  areas  other  than  defects  and  program  measures  (members  of  the 
group  did  not  have  any  examples  on  this) 

•  selecting  measurements  that  are  relevant  to  control  charts,  and  that  when  plotted  yield 
meaningful  results 

4.3  Working  Group  2:  Assessments 

Participants:  A1  Aldrich,  Mark  Amaya,  Julie  Barnard,  Colin  Benton,  Harry  Carl,  Donna 
Dunaway,  Judah  Mogilensky,  Jane  Moon,  Joe  Puffer,  V.  Subramanyam,  Carolyn  Swanson, 
Don  White,  and  Gary  Wigle. 

Jane  Moon  (Raytheon)  facilitated  the  brainstorming  of  the  Assessment  Working  Group  in 
order  to  collect  issues  that  the  group  felt  were  the  most  important: 

•  consistency  of  assessment  results 
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•  requirements  for  assessments 

-  training  for  the  team  (#3) 

-  assessment  team  members  (#3) 

-  lead  assessors  (#3) 

-  organizational  level  4  understanding  of  quantitative  management  (Curtis  question  8) 

(#1) 

•  Is  SPC  necessary? 

•  difference  between  quantitative  techniques,  statistical  control,  and  SPC? 

•  number  of  subprocesses  that  need  to  be  controlled 

•  training  for  level  4-5  (#3) 

•  Institutionalization?  length  of  time 

•  Do  processes  (under  SPC)  have  to  be  capable? 

•  For  TCM,  what  constitutes  “new  technology?” 

•  Is  piloting  required? 

•  Must  use  QPM/SQM  for  PCM/TCM?  (And  if  not,  what’s  the  difference  between 
PCM/TCM  and  OPF/OPD  at  Level  3?) 

•  Does  SPC  require  use  of  control  charts? 

•  What  project  characteristics  are  required  to  institutionalize  Level  4? 

•  Do  we  require  a  clear  link  from  business  goals  to  measurements  (selected  processes)? 

•  What  constitutes  “analyzed”  below  Level  4? 

•  How  many  types  of  measurements  are  required  for  Measurement  and  Analysis  common 
feature  (at  any  level)? 

•  What  is  a  “measurement”? 

•  What  are  “minimum  criteria”  for  Level  4? 

•  What  are  quality  criteria  under  SQM? 

•  What  defined  process  is  required  for  PCM  and  TCM? 

•  What  vehicles  exist  for  disseminating  CMM  interpretations/guidance? 

At  the  conclusion  of  the  brainstorming,  the  group  considered  the  question  of  what  they 
wanted  to  accomplish.  Did  they  want  to  focus  on  infrastructure  or  focus  on  making  group 
interpretations?  The  goals  of  this  session  were  determined  to  be 

•  developing  guidelines  on  Level  4-5  interpretation.  (All  group  members  wanted  to  do  this) 

•  creating  mechanism  for  establishing  and  disseminating  guidelines  (all  but  three) 

-  to  Lead  Assessors 

-  to  organizations  working  on  Level  4-5 

•  establishing  training  and  qualification  of  Level  4-5  Lead  Assessors  and  teams  (all  but 
two) 
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The  group  agreed  to  begin  discussions  on  the  second  bullet  “mechanism  for  establishing  and 
disseminating  guidelines.”  The  discussion  points  included 

•  Re-establish  CMM  Advisory  Board. 

-  mechanism  for  community  ownership  of  model 

•  Establish  accreditation  criteria  for 

-  Lead  Assessors 

-  appraised  organization 

•  Document  this  group’s  interpretation  and  send  it  out  to  everyone. 

Copies  of  “SEI  Strategy  for  Ensuring  Valid  Implementation  and  Appraisal  of  Level  4  and 
Level  5  Process  Areas  -  October  28,  1999”  were  distributed.  The  document  is  attached  in 
Appendix  C.  The  group  had  a  number  of  revisions  that  they  wanted  to  recommend. 

•  Re-establish  CMM  Advisory  Board.  (Add  to  paragraph  1 .) 

•  Establish  mandatory  supplemental  training  of  any  Lead  Assessor  (to  lead  Level  4-5  as¬ 
sessments).  (Add  to  paragraph  1 .) 

•  Gather  data  re/  “problem  of  inconsistent  results  at  Level  4-5.”  (Add  to  paragraph  1.) 

•  Strengthen  QA  provisions  for  Lead  Assessors.  (Add  to  paragraph  1.) 

•  Periodically  conduct  High  Maturity  practices  workshop.  (Add  to  paragraph  5.) 

•  Elicit  papers  from  the  community  at  large.  (Add  to  paragraph  6.) 

•  Identify  criteria  for  qualified  referees.  (Add  to  paragraph  6.) 

•  Add  Report  of  the  Workshop  Proceedings  and  mandate  to  grow  further.  (Add  to  para¬ 
graph  7.) 

•  Drop  the  word  “informal”  from  title  of  paragraph  8.  Re-title  “Communications  between 
the  SEI  and  the  CMM  User  Community.” 

After  a  short  break,  the  group  moved  to  training  and  qualifications  of  Lead  Assessors  and 
teams.  Items  discussed  included 

•  One-day  Level  4-5  training  will  be  developed  to  supplement  CMM  model  training  (re¬ 
flects  CMM  v  1.1)  (add  to  paragraph  3. c.) 

•  Courses  to  be  offered  by  SEI  need  final  peer  review  by  qualified  individuals  in  the  com¬ 
munity  (e.g.,  by  CMM  Advisory  Board  or  designees).  (Add  to  paragraph  3.) 

•  “Continuing  education”  criteria  and  expectations  will  be  established  for  Lead  Assessors 
(for  renewal  of  LA  authorization).  (Add  to  paragraph  4.) 

•  Require  high  maturity  practices  course  (or  defined  equivalent)  for  Lead  Assessors  to  lead 
L4-5.  (Add  to  paragraph  4.) 

•  Require  Lead  Assessors  for  L4-5  assessments  to  have  experience  with  assessments  where 
L4-5  KPAs  are  in  scope.  (Add  to  paragraph  4.) 
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•  All  requirements  described  for  Lead  Assessors  apply  to  Lead  Evaluators  as  well.  (Add  to 
paragraph  4.) 

•  Separate  method  training  from  model  training.  (Add  to  paragraph  4.) 

The  following  recommendations  were  agreed  upon: 

•  Make  materials  available  through  Transition  Partners. 

•  Provide  training  in  other  geographic  areas  (e.g.,  Middle  East,  India,  Australia,  etc.). 

The  discussion  moved  to  bullet  one  above:  guidelines  on  Level  4-5  interpretation.  Focus  on 
SPC  and  “acceptable  limits.” 

•  SPC  includes  many  techniques — and  IS  required  for  satisfying  Level  4. 

•  statistical  methods,  including  Pareto, . . .,  etc. 

•  control  charts  (or  “process  behavior”  charts) 

•  limits  established  by  “voice  of  the  process”  (stable)  vs.  “voice  of  the  customer”  (capable) 

•  other  approaches:  multi-variety  analysis 

•  NOT  arbitrarily  set  limits  that  are  not  process  data  driven 

•  based  on  a  theoretical  foundation 

•  finer  granularity  than  Level  2 

•  Is  there  something  I  can  change  to  get  this  process  under  control? 

•  Techniques  applied  to  manufacturing  do  not  necessarily  apply  directly  to  SW  engineer¬ 
ing. 

•  Does  it  meet  the  needs  to  the  project? 

•  Is  there  a  coherent  rationale? 

•  QPM  requires  identification  of  critical  subprocesses  that  are  placed  under  statistical  con¬ 
trol. 

-  the  understanding  of  the  natural  variation  of  the  subprocesses  (“voice  of  the  process”) 

-  Take  action  for  “special  causes  of  variation”  (once  you  can  recognize  them). 

-  Limits  are  established  by  the  “voice  of  the  process”  (stable). 

-  aware  of  process  vs.  voice  of  the  customer 

•  NOT  arbitrarily  set  limits  that  are  not  derived  from  the  natural  variation  of  the  process 
(i.e.,  based  on  the  natural  performance  of  the  process) 

•  SPC  includes  many  techniques,  and  it  is  expected  that 

-  Techniques  applied  to  manufacturing  do  not  necessarily  apply  directly  to  software 
engineering. 

-  Control  charts,  or  “process  behavior”  charts  are  expected  at  a  minimum. 

-  Statistical  methods,  including  Pareto  charts,  histograms,  etc.,  are  included 

-  Other  approaches  may  be  used:  multivariate  analysis. 
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Quantitative  management  methods  are  in  place  to  address  issues  of  instability  (that  is, 
managers  use  the  data  to  make  decisions). 


Next  topic:  Require  a  link  between  the  business  goals  and  the  measurements 


•  related  to  the  selected  subprocesses 

•  Processes  help  achieve  identified  business  goals. 

Next  topic:  Number  of  subprocesses  to  be  controlled 

•  critical  subprocess,  as  applicable  to  your  issues/goals 

—  Use  (sensitivity)  analysis  to  establish  criteria  for  selecting  “critical  subprocesses. 

-  Not  all  processes/subprocesses  need  to  be  under  quantitative  control. 

Next  topic:  PCM  and  TCM— Is  piloting  required? 

•  limited  in  scope 

•  applied  when/where  appropriate 

•  Some  examples  of  piloting  are  observed  (in  assessment). 

•  Expect  to  see  piloting  (or  prior  use)  before  major  changes. 

•  Organization  has  defined  criteria  for  piloting. 

4.4  Working  Group  3:  CMM  Integration 

Participants:  Dottie  Acton,  Julie  Barnard,  Rick  Biehl,  Mary  Lynn  Penn,  Neil  Potter,  Warren 
Schwomeyer,  and  Ramesh  Venkatraman. 

A  collaboration  of  government,  industry  and  the  Software  Engineering  Institute  is  integrating 
the  Capability  Maturity  Models  (CMM)  with  an  objective  of  making  the  CMMI  models  an 
international  standard. 

Participants  in  the  High  Maturity  Practices  Workshop  of  November  1999  discussed  various 
issues  pertaining  to  CMM  Integration.  Mary  Beth  Chrissis,  SEI,  facilitated  the  discussion, 
and  the  participants  were  from  matured  organizations  that  are  all  following  the  CMMI  prac¬ 
tices.  Refer  to  the  annexure  for  participating  organizations’  names. 

The  participants  listed  out  the  points  for  discussion  and  all  the  stated  points  were  voted  for 
priority.  Based  on  the  voting,  the  following  items  were  prioritized  and  the  report  is  summa¬ 
rized. 

•  CMM  Global  Issues 
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•  impact  of  the  CMMI 

•  continuous  v/s  staged 

•  statistical  v/s  quantitative 

•  things  to  worry  about 

•  normative  v/s  Informative 


The  participants  also  presented  the  gist  of  the  discussions  during  the  workshop,  and  this  re¬ 
port  is  the  collective  work  of  the  working  group  submitted  to  the  SEI.  This  report  expresses 
the  concerns  and  the  suggestions  for  improvement  with  respect  to  CMM  Integration,  with  an 
intention  of  what  shall  be  done  further  by  the  SEI  on  CMM  Integration  for  the  benefit  of 
practicing  organizations. 

4.4.1  Impact  of  CMMI 

The  Integrated  Capability  Maturity  Model  (CMMI)  is  an  excellent  representation  of  both 
Software  and  Systems  Engineering  practices.  Industry,  as  it  has  matured  in  one  area,  has 
consistently  defined  weaknesses  in  associated  areas.  The  partnership  of  the  two  disciplines  is 
both  necessary  and  welcome. 

Industry,  however,  has  adopted  other  models  to  guide  improvement  in  associated  areas. 

These  areas  include  Human  Resources/Training  (People  CMM),  Subcontract  Selection  and 
Management  (Software  Acquisition  CMM),  and  the  increased  need  to  adopt  IPPD.  In  order 
to  assure  industry  that  the  CMMI  is  indeed  “THE”  model,  a  plan  should  be  made  public  with 
details  of  further  model  integration. 

Particular  impacts  to  industry  were  also  enumerated;  some  of  these  are  discussed  in  more 
detail  in  other  sections; 

1.  Peer  Review  Key  Process  Area  (KPA):  In  the  past  versions  of  the  SW  CMM,  Peer  Re¬ 
view  was  called  out  as  a  stand-alone  KPA.  It  was  this  KPA  which  gave  much  of  the 
foundation  as  maturity  progressed.  Forcing  the  introduction  of  a  formal  peer  review  at 
Level  2  assisted  organizations  in  establishing  quality  goals,  associated  metrics  and  pre¬ 
vention  activities.  The  incorporation  of  peer  reviews  into  generic  review  processes  will 
diminish  the  importance  of  the  practice  itself. 

2.  Measurement  and  Analysis  Process  Area  (M&A  PA):  In  the  past  versions  of  the  models, 
measurements  had  been  a  Key  practice  associated  with  every  process  area.  The  focus  of 
measurement  and  analysis  as  a  separate  Process  Area  gives  importance  and  concentra¬ 
tion  to  the  activity  and  lays  a  stronger  foundation  for  improvement  and  quantitative 
management. 

3.  Risk  Management  PA:  The  addition  of  this  process  area  was  a  strong  indication  of  the 
added  importance  of  quantitatively  managing  the  process.  Although  the  PA  is  very 
complete  in  discussing  the  relevance  and  mitigation  of  risk,  it  is  noticeably  silent  on  the 
managing  opportunities. 
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4.  Quantitative  Process  Management:  The  new  model  clearly  articulates  the  meaning  of 
quantitatively  managing  as  it  relates  to  quantitative  control. 

5.  CMMI  introduction:  There  is  the  obvious  tendencies  in  the  beginning  sections  to  “de¬ 
fend”  versus  introduce  the  model.  There  is  an  overload  (63  pages)  of  “apple  pie”;  for 
the  novice  user  this  would  not  solicit  the  adoption  of  the  model. 

6.  Continuous  versus  Staged:  Although  this  will  be  discussed  in  later  sections,  the  general 
opinion  was  that  educating  the  user  in  the  CMMI  Process  Areas  and  the  need  to  establish 
good  practices  is  a  significant  challenge.  Educating  the  user  in  the  benefits  of  continu¬ 
ous  versus  staged  is  overwhelming.  One  representation  needs  to  be  adopted. 

7.  General  Format:  The  CMMI  program  needs  to  make  industry  aware  that  the  “packag¬ 
ing”  was  dictated  by  the  desire  to  adopt  the  CMMI  as  an  international  industry  standard. 

8.  Normative  versus  Informative  Model:  This  topic  also  is  discussed  in  later  sections.  The 
normative  model  is  an  excellent  representation  from  the  assessor’s  point  of  view.  The 
number  of  work  aids  included  in  the  model  caused  confusion  and  was  over  burdening. 
Separating  the  normative  and  informative  emphasized  the  difference. 

It  is  important  for  industry  to  understand  that  there  are  issues  and  differences.  Encourage¬ 
ment  to  embrace  the  model  will  come  with  evidence  of  its  usefulness  and  ease  of  adoption. 

4.4.2  Things  to  Worry  about  for  Level  4  and  5  Organizations 
Adopting  the  CMMI 


This  worry  list  stems  from  a)  scope  changes  found  in  CMMI,  and  b)  the  interpretation  prob¬ 
lems  likely  to  arise  for  an  assessor. 

Level  2:  Data  Management 

Data  management  could  be  interpreted  by  an  assessor  as  the  management  of  program  and 
project  data  under  Configuration  Management.  In  this  case,  there  is  no  worry.  It  could  also  be 
interpreted  as  the  management  of  data  that  you  have  never  managed  before.  That  could  cause 
a  surprise  in  an  assessment. 

Level  2:  Supplier  Agreement  Management 

This  Process  Area  now  includes  all  significant  products  and  services  acquired  to  make  your 
projects  successful.  There  is  much  room  for  interpretation  (and  therefore  room  to  be  assessed 
Level  1 !).  Read  the  CMMI  carefully,  and  check  the  following  with  your  assessor: 

•  Does  the  assessor  interpret  “Supplier”  to  mean  only  those  products  you  purchase  that  are 
delivered  to  your  customer?  Or  is  “Supplier”  interpreted  to  mean  all  products  and  serv¬ 
ices  purchased  for  your  project,  regardless  of  whether  or  not  they  are  delivered  to  your 
customer?  In  the  Intro  to  CMMI-0.2  class,  the  SEI  instructors  did  not  agree  on  this  issue. 

•  How  does  your  assessor  define  “service”?  Assessors  might  differ  greatly  in  their  inter¬ 
pretations  of  what  a  service  is,  and  which  ones  come  under  Supplier  Agreement  Man- 
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agement.  I  have  not  found  a  good  definition  of  “service”  in  the  CMMI  yet.  The  SEI 
gives  examples  of  product  help-desk  support,  post-delivery  product  consulting,  and  cus¬ 
tomer  training. 

Level  2:  Project  Planning  and  Control  Process  Areas 

The  process  areas  state  that  products  and  services  need  to  be  planned  and  tracked.  “Services” 
might  include  product  help-desk  support,  post-delivery  product  consulting,  and  customer 
training.  If  an  assessor  has  this  interpretation,  these  business  functions  will  be  assessed.  Are 
yours  Level  2/3/4/5? 

Level  3:  Decision  Analysis  and  Resolution 

This  process  area  requires  the  need  for  a  generic  decision  analysis  process  for  making  im¬ 
portant  decisions.  If  you  do  not  formally  have  one  of  these,  you  would  be  assessed  at  Level  2. 

Assessing  Combined  Systems  Engineering  and  Software  Engineering  Groups 

Some  assessors  might  insist  that  both  systems  and  software  engineering  groups  be  assessed  in 
one  assessment,  since  the  CMMI  naturally  covers  both  disciplines.  However,  the  systems 
group  might  be  at  Level  2,  and  the  software  group  at  Level  4.  A  combined  assessment  might 
lead  to  difficulty  in  determining  prevalent  strengths  and  weaknesses. 

Make  sure  your  assessor  understands  the  correct  assessment  scoping  rules  currently  present 
in  the  CBA-IPI  method,  (and  whatever  is  in  the  upcoming  SCAMPI  method).  CBA-IPIs  can 
scope  the  assessment  in  many  ways,  as  long  as  this  scope  is  clear  in  the  findings. 

Continuous  View 

Level  5  in  the  staged  view  is  not  the  same  as  Level  5  in  the  continuous  view.  Level  4/5  in  the 
continuous  view  requires  that  the  measurements  of  all  the  process  areas  be  under  statistical 
control.  This  can  be  meaningless  for  some  process  areas,  such  as  CM  and  Risk  Management. 

4.4.3  CMMI  -  Normative  versus  Informational  Models 

Each  representation  of  the  CMMI  (continuous  and  staged)  has  two  models:  the  Normative 
and  the  Informational.  The  discussion  centered  on  understanding  the  differences  between  the 
models  and  identifying  some  of  the  concerns  associated  with  one  or  the  other  of  the  models 
or  with  having  two  models. 

There  was  a  general  concern  expressed  regarding  the  sheer  volume  of  material  associated 
with  the  CMMI.  Obviously  one  contributing  factor  is  the  presence  of  both  a  Normative  and 
Informational  Model  for  each  representation  of  the  CMMI.  The  inclusion  of  the  Normative 
Model  within  the  Informational  Model  also  adds  to  the  total  volume  of  material.  In  contrast 
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to  the  recommendation  to  have  only  one  CMMI  representation,  the  group  did  not  reach  con¬ 
sensus  on  the  desirability  of  having  only  one  CMMI  model. 

The  SEI  provided  insight  into  the  origin  of  the  Normative  Model.  The  packaging  of  this 
model  is  influenced  by  the  need  to  support  making  the  CMMI  an  international  standard.  As 
evidenced  by  EIA/IS  731,  an  international  standard  contains  considerably  less  detail  than  the 
Informational  Model.  To  satisfy  this  need,  a  recommendation  was  made  to  the  SEI  to  reduce 
the  size  of  the  Normative  Model  even  more  (to  20  pages),  call  it  a  Standard  and  provide  ref¬ 
erence  to  the  Informational  Model  for  all  of  the  descriptive  material  associated  with  under¬ 
standing  and  interpreting  the  standard. 

The  higher  level  of  abstraction  provided  by  the  Normative  Model  was  viewed  to  have  both 
benefits  and  drawbacks  for  low  maturity  organizations.  On  the  one  hand,  it  was  felt  this  ab¬ 
straction  would  help  prevent  low  maturity  organizations  from  ‘nit  picking’  the  model  and 
getting  bogged  down  in  minutia  in  one  Process  Area,  or  at  one  level,  while  ignoring  other 
practices  which  would  have  a  more  beneficial  effect  on  their  operations.  At  the  same  time,  it 
was  felt  that  a  low  maturity  organization  could  miss  the  beneficial  points  of  a  Process  Area 
described  in  the  Informational  Model  or  use  this  abstraction  to  intentionally  get  away  with 
less  rigor  through  “interpretation”  of  the  Normative  Model. 

This  perspective  that  the  Normative  Model  is  “watered  down”  and  “more  abstract”  when 
compared  with  the  Informational  Model  raised  questions  as  to  whether  one  might  be  too  little 
and  the  other  too  much.  A  contributor  to  this  perspective  may  be  related  to  the  use  of  bold 
type  to  indicate  the  required  portions  of  the  model.  The  textual  wording  of  the  practices  is 
bold  type  in  the  Informational  Model  and  regular  type  in  the  Normative  Model  indicating  a 
difference  in  the  requirements.  This  leads  to  the  conclusion  that  the  practices  in  the  Norma¬ 
tive  Model  are  at  a  higher  level  than  in  the  Informational  Model.  If  there  is  to  be  one  opera¬ 
tive  model,  it  must  be  more  clearly  communicated  how  compliance  to  either  of  the  two 
documents  should  result  in  similar  and  equally  mature  processes. 

Related  to  the  concern  of  implementing  a  more  or  less  abstract  model  is  the  concern  of  being 
assessed  to  a  more  or  less  abstract  model.  Questions  raised  included 

•  Will  assessment  to  the  Normative  Model  be  more  subjective  with  respect  to  assessing  to 
differing  interpretations  of  the  goals  and  practices? 

•  Will  assessment  to  the  Informational  Model  be  “nit  picking”’  down  to  the  information  in 
the  sub  and  sub-sub-practices? 

•  Will  assessors  be  given  enough  training  and  guidance  to  perform  consistent  assessments 
irrespective  of  the  model  used? 

Consistency  of  assessment  results  is  critical  to  the  effective  use  of  the  CMMI  (as  with  any 
maturity  model).  It  was  indicated  that  it  is  necessary  for  organizations  to  provide  feedback  to 
the  SEI  about  their  assessments  and  assessors  in  order  to  help  ensure  consistent  assessments. 
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The  SEI  indicated  to  the  group  that  there  are  currently  inconsistencies  in  the  level  of  detail  in 
defining  some  of  the  practices  in  the  Informational  Model  due  to  adhering  to  schedule  con¬ 
straints  in  releasing  the  CMMI.  The  next  release  of  the  model  is  expected  to  correct  these 
inconsistencies. 

4.4.4  Continuous  versus  Staged 

This  topic  addressed  the  relative  advantages  and  disadvantages  of  the  two  different  model 
implementations,  concerns  with  the  differences  between  the  models,  and  recommended  that 
the  SEI  adopt  one  or  the  other,  but  not  both. 

Advantages  and  Disadvantages 

One  advantage  of  the  staged  model  is  that  it  is  easy  to  get  started  with,  because  people  can 
box  themselves  at  a  level  and  work  toward  one  level  at  a  time.  Of  course,  this  has  a  corre¬ 
sponding  disadvantage  that  people  will  look  only  at  the  current  level  rather  than  trying  to 
gain  an  up-front  understanding  of  the  entire  model.  This  up-front  understanding  would  influ¬ 
ence  their  implementations  at  lower  levels  in  order  to  make  the  higher  levels  more  easily 
achievable. 

The  corresponding  advantage  of  the  continuous  model  is  that  it  encourages  looking  across  the 
entire  model  initially,  which  facilitates  the  creation  of  a  firmer  foundation  at  lower  levels.  It 
also  encourages  an  organization  to  think  about  what  its  issues  are  and  pick  those  areas  that 
are  most  relevant  to  its  organizational  issues  and  concerns.  A  mapping  is  available  that 
shows  which  areas  to  pick  in  order  to  parallel  the  staged  model,  but  there  is  no  requirement  to 
pick  that  set,  or  to  implement  all  practices  in  a  category  at  one  time.  This  allows  a  more  natu¬ 
ral  progression  of  maturity  across  each  process  area  based  on  business  needs. 

Concerns 

The  most  obvious  problem  of  the  continuous  model  is  that  it  breaks  down  at  Levels  4  and  5 
because  of  the  application  of  Level  4  and  5  generic  practices  to  each  process  area.  With  the 
continuous  model,  in  order  to  achieve  a  Level  5  rating  in  each  process  area,  the  processes  for 
each  area  must  be  quantitatively  managed,  have  quantitative  improvement  objectives  and  be 
measurably  improved.  Contrast  this  with  the  staged  model,  where  the  emphasis  is  on  select¬ 
ing  the  processes  that  can  benefit  most  from  quantitative  management  and  improvement. 

An  examination  of  the  current  Level  5  organizations  might  make  this  concern  clearer.  How 
many  of  the  current  Level  5  organizations  have  some  or  all  of  their  SQA,  SCM,  SSM,  OPF 
and  OPD  processes  under  quantitative  management?  This  would  be  required  for  a  Level  4 
rating  against  the  continuous  model,  whereas  with  the  staged  model,  most  Level  5  organiza¬ 
tions  have  selected  to  quantitatively  manage  and  improve  only  some  processes  (inspection, 
testing,  requirements  management,  etc),  not  processes  from  every  process  area. 
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Another  area  of  concern,  other  than  the  assessment  issues,  is  that  the  two  models  do  not  ap¬ 
pear  to  be  equivalent.  There  are  more  practices  in  the  continuous  model  since  the  level  1 
practices  and  higher  level  practices  do  not  show  up  in  the  staged  model.  For  example,  in  the 
Customer  and  Product  Requirements  process  area,  there  are  10  activities  in  the  staged  model, 
and  12  specific  practices  in  the  continuous  model.  The  first  difference  is  that  the  staged 
model  has  a  Level  1  specific  practice,  Collect  Stakeholder  Needs,  which  is  not  in  the  staged 
model.  However,  the  Elicit  Needs  practice  and  other  related  practices  are  in  both  models,  so 
this  is  not  too  troublesome.  The  second  difference,  however,  is  more  significant.  The  con¬ 
tinuous  model  has  a  Level  4  practice,  Perform  Quantitative  Validation  of  Customer  Require¬ 
ments,  which  is  not  in  the  staged  model.  This  is  particularly  troublesome,  because  it  is  yet 
another  way  in  which  a  Level  4  rating  against  the  continuous  model  is  more  stringent  than  a 
Level  4  rating  against  the  staged  model. 


Summary  and  Recommendation 

By  a  9  to  1  vote,  the  group  preferred  the  continuous  model.  However,  the  caveat  associated 
with  this  is  that  the  requirements  for  assessment  at  the  higher  levels  must  be  clarified,  or  it 
must  be  made  explicit  that  the  bar  is  being  raised  as  to  what  it  means  to  be  a  Level  4  or  5  or¬ 
ganization  when  using  the  continuous  model. 

By  a  unanimous  vote,  the  group  preferred  their  second  choice  of  model  to  having  two  mod¬ 
els.  It  was  felt  that  the  complexity  and  confusion  introduced  by  having  to  choose  between 
models  and  to  reconcile  ratings  against  two  models  was  not  balanced  by  the  advantages  of 
either  model. 

4.4.5  Statistical  versus  Quantitative 

The  working  group  discussed  a  distinction  that  they  felt  was  often  being  missed  when  dis¬ 
cussing  higher  levels  of  process  maturity;  namely,  the  distinction  to  be  drawn  between  quan¬ 
titative  techniques  generally  and  statistical  techniques  specifically.  The  working  group 
agreed  that  the  latter  group  represents  a  subset  of  the  former,  but  agreed  that  virtually  any 
discussion  of  data  at  higher  maturity  levels  seems  to  concentrate  only  on  the  narrower  statis¬ 
tical  category. 

While  a  more  detail  discussion  of  the  various  techniques  was  presumed  to  be  going  on  in  the 
various  measurement  working  groups,  this  working  group  chose  to  concentrate  on  the  re¬ 
quirements  for  using  this  distinction,  and  how  such  requirements  are,  or  should  be,  reflected 
in  the  CMMI. 

Relevant  requirements  that  were  elicited  during  the  discussion  concentrated  on  the  purposes 
to  which  data  was  being  put  in  the  process  improvement  arena,  allowing  such  purposes  to 
drive  the  choice  from  among  the  various  quantitative  techniques. 
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While  the  majority  of  participants  in  the  group  were  using  SPC  in  their  organizations,  all 
agreed  that  an  organization  should  be  concentrating  on  statistical  control  only  when  rigor  is 
required  beyond  that  which  can  be  achieved  through  less  formal  quantitative  techniques. 
Specifically,  more  rigor  is  required  when  emphasizing  long-term  systemic  change,  or  when 
the  emphasis  is  on  reducing  variance  rather  than  shifting  the  mean,  and  when  situations  arise 
where  change  agents  strongly  feel  that  outliers  are  heavily  influencing  analysis. 

All  agreed  on  the  importance  of  measurement  and  analysis  rigor,  and  all  agreed  that  statistics 
generally,  and  SPC  specifically,  can  and  should  be  used  to  drive  such  rigor.  But  the  group 
also  acknowledged  that  many  lower  level  maturity  organizations  view  statistics  as  a  “wall” 
that  needs  to  be  scaled  in  order  to  improve  maturity.  The  shift  to  statistics  is  not  viewed  as 
smooth,  and  can  be  an  inhibitor  to  organizations  choosing  to  progress  toward  higher  maturity 
levels.  There  is  also  considerable  concern  over  the  criteria  assessors  should  use  when  evalu¬ 
ating  an  organization's  use  of  these  techniques. 

The  group  observed  that  the  CMMI  tends  to  describe  quantitative  requirements  that  are  al¬ 
most  universally  interpreted  by  readers  as  statistical  requirements.  The  distinction  should  be 
better  drawn  in  future  versions  of  the  model.  Normative  descriptions  should  emphasize  re¬ 
quirements  for  quantitative  analysis,  while  informative  descriptions  might  cite  more  and 
deeper  statistical  examples  and  applications. 

Higher  maturity  organizations  typically  have  had  success  with  statistical  techniques.  Under¬ 
standably,  they  want  to  encourage  lower  maturity  organizations  to  adopt  these  techniques  as 
part  of  their  process  improvement  activities.  However,  such  enthusiasm  must  be  tempered 
with  the  realities  and  how  such  movement  is  perceived  by  lower  maturity  level  organizations. 
Applied  too  soon,  to  unreliable  low  maturity  data,  statistics  can  be  time-consuming  and 
counterproductive.  The  CMMI,  with  its  emphasis  on  applications  of  all  process  areas  across 
the  maturity  spectrum,  must  recognize  and  support  this  distinction. 

4.4.6  Theory  versus.  Practice 

The  working  group  observed  that  the  CMMI  represents  a  vast  expansive  model  when  com¬ 
pared  to  the  previous  software  CMM.  This  expansion  discussed  is  qualitatively  different  than 
the  simple  expansion  of  scope  brought  about  with  the  integration  of  software  and  system  en¬ 
gineering  models.  The  distinction  discussed  is  inherent  in  any  shift  from  staged  to  continu¬ 
ous,  as  discussed  previously. 

By  shifting  to  a  continuous  model,  the  user  of  the  CMMI  is  theoretically  responsible  to  apply 
all  general  practices  to  all  process  areas.  In  practice,  this  will  not  be  possible  for  all  users, 
particularly  lower  maturity  organizations.  Higher  maturity  users  of  the  SW-CMM  have  little 
difficulty  acknowledging  and  describing  how,  in  theory,  the  staged  model  can  be  interpreted 
as  continuous.  Every  KPA,  regardless  of  the  maturity  level  to  which  it  has  been  defined,  can 
be  applied  at  every  maturity  level.  In  practice  however,  presenting  the  SW-CMM  as  a  staged 
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model  made  such  interpretation  unnecessary  for  lower  maturity  organizations  unable  to  see 
such  distinctions  for  themselves. 


The  group  discussion  resulted  in  a  core  theme;  namely,  does  the  continuous  model  actually 
work  in  theory,  or  might  its  application  lead  to  unintended  negative  consequences  in  practice? 
An  example  discussed  at  length  was  the  Quantitatively  Manage  Process  Performance  prac¬ 
tice.  Can  (or  should)  this  be  done  practically  for  every  process  area,  or  is  it  only  theoretically 
possible?  The  group  did  not  try  to  pursue  the  question  to  a  complete  answer,  rather  focusing 
on  the  role  that  the  CMMI  model  can  or  should  play  in  providing  such  an  answer. 

The  issue  carried  weight  among  group  members  because  of  the  assessment  implication  of  the 
implied  choices.  Participants  acknowledged  that  often  such  practice-level  choices  are  made 
simply  to  remain  competitive  with  other  organizations  that  are  perceived  to  be  conducting 
certain  practice  in  certain  ways.  This  decision  environment  requires  better  informational 
support  from  the  CMMI  than  was  required  under  the  old  model. 

The  group  recommends  that  the  CMMI  be  developed  to  include  more  extensive  work  aids  to 
provide  meaningful  paths  through  the  model;  aiding  some  of  the  theory-to-practice  decisions 
that  must  be  made.  The  group  acknowledged  that  such  information  needs  to  arise  from  the 
community  of  users,  and  that  the  SEI  is  not  in  a  position  to  make  such  in-practice  decisions 
for  the  user  community.  However,  the  SEI  can  be  a  key  facilitator  in  moderating  such  a  flow, 
and  assuring  that  the  theory-practice  decisions  being  made  in  the  user  community  are  syn¬ 
chronized  with  the  expectations  of  the  assessment  process.  Without  such  synchronization, 
poor  decision  making  is  likely  to  follow  even  in  higher  maturity  organizations. 

4.5  Working  Group  4:  Measurement  (Session  A) 

This  session  included  the  following  participants: 

•  Bhaskar  Chavali,  NUT 

•  Ellen  George,  Allied  Signal 

•  Kelly  Gunning,  Marconi  Integrated  Systems 

•  Rick  Hefner,  TRW 

•  Barbara  Hirsh,  Motorola 

•  Bijay  Kumar  Jyotishi,  WIPRO  Systems  Ltd. 

•  Barbara  Kolkhorst 

•  Andy  Meadow,  PRC 

•  Michael  Scott,  Raytheon 

•  Charlie  Weber,  SEI 

A  variety  of  measurement-related  topics  were  identified,  but  two  were  selected  as  most  im¬ 
portant.  Discussions  follow. 
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4.5.1  Linking  Business  Goals  to  Project  Goals  and  Measures 


One  of  the  difficulties  high  maturity  organizations  face  is  the  linkage  between  high  level 
business  goals  and  the  project  level  goals  and  measures,  and  how  software  fits  in  to  help 
achieve  the  goals.  The  difficulties  begin  with  the  definition  of  the  business  goals  themselves, 
and  how  they  are  interpreted  down  through  the  organization  to  the  projects  and  individuals. 

Business  goals  are  often  defined  or  created  by  high  level  management  and  tend  to  be  more  of 
slogans  rather  than  goals.  They  are  very  often  not  stated  in  a  quantifiable  way  (e.g.,  “Im¬ 
prove  Shareholder  Value,”  “Increase  Market  Share,”  “Employee  Satisfaction,”  “Customer 
Satisfaction”).  Without  quantifiable  goals  it  is  difficult  to  assess  success  against  the  goal. 

This  can  lead  to  different  interpretations  of  the  goal  so  that  the  resulting  lower  level  goals 
become  widely  varied. 

Another  problem  with  some  business  goals  is  that  they  are  not  grounded  by  realism.  Stretch 
goals  are  a  good  thing,  but  there  must  be  some  chance  of  success  or  there  will  be  no  buy-in  in 
the  organization.  Goals  that  are  obviously  unachievable  are  identified  easily  by  personnel  in 
the  organization  and  can  actually  result  in  not  being  able  to  achieve  any  of  the  goals  due  to 
credibility.  Additionally,  unrealistic  goals  at  the  organization  level  often  result  in  unrealistic 
project  goals.  Business  goals  should  be  based  on  the  capability  of  the  organization.  One  of 
the  practices  that  help  achieve  this  is  the  Balanced  IT  Scorecard  Method,  available  at  <URL: 
http://www.esi.es/Publications/Reports/ME99TR014.html>,  that  brings  all  of  the  stakeholders 
into  the  goal  setting  process. 

Project  level  goals  must  be  mapped  to  the  organizational  goals  and  must  be  quantifiable,  and 
achievable  (even  if  organization  members  must  stretch  to  achieve  them).  One  of  the  difficul¬ 
ties  projects  have  in  establishing  their  goals  is  the  short-term  view  of  the  project  (e.g.,  get  the 
product  out  in  the  given  timeframe)  versus  the  long-term  organizational  view  (e.g.,  business 
growth).  This  tactical  versus  strategic  conflict  can  sometimes  result  in  conflicting  goals  that 
must  be  acknowledged  and  accounted  for,  if  not  eliminated.  A  practice  that  has  been  success¬ 
ful  in  establishing  project  goals  is  the  Goal,  Question,  Metric  (GQM)  paradigm,  available  at 
<URL:  http://www2.umassd.edU/SWPI/ESEG/localmat.html#gqm>.  In  this  method  we  es¬ 
tablish  a  goal,  question  whether  the  goal  is  adequate,  quantifiable,  achievable,  linked  to 
higher  level  goals,  and  then  establish  a  measure  or  metric  to  determine  success  against  the 
goal. 

The  final  issue  that  is  related  to  this  topic  is  the  linkage  of  incentives  (both  positive  and 
negative)  to  the  goals  at  each  level.  Without  incentives  the  question  will  always  be  “Why 
should  I  care  about  this  goal?  Incentives  can  take  form  at  all  levels  (e.g.,  organizational, 
project,  team,  individual)  and  should  be  timely.  One  organization’s  practice  is  a  formula  for 
annual  bonuses  that  are  based  on  individual  performance,  team  performance,  organizational 
performance,  and  partnering  with  other  organizations  in  the  company. 
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The  establishment  of  meaningful  goals  at  the  project  level  for  products,  processes,  or  indi¬ 
viduals  starts  with  the  definition  of  the  higher  level  organization  or  business  goals.  These 
goals  must  be  quantifiable  and  achievable,  and  must  then  flow  through  the  organization. 
Links  must  be  made  between  the  business  goals  and  the  project  goals  and  measures  put  in 
place  to  assess  success.  All  of  this  requires  an  enlightened  senior  management  structure  that 
is  willing  to  let  the  organization  help  in  establishing  the  business  goals.  There  are  many  best 
practices  being  used  in  high  maturity  organizations,  but  there  is  nothing  standard  across  the 
industry. 

4.5.2  Similarities  and  Differences  in  People  Issues  between 
Lower  and  Higher  Maturity  Levels 

It  was  the  consensus  of  the  group,  after  discussion,  that  there  are  behavioral  clues  that  are 
indicative  of  the  maturity  level  of  the  organization.  That  is,  organizations  of  different  matur¬ 
ity  levels  behave  differently.  The  following  ideas  were  developed. 

There  are  differences: 

•  Buy-in  is  a  continuous  process.  But  buying  in  to  CMM  SW  Levels  2  and  3  does  not 
automatically  lead  to  buy-in  for  Levels  4  and  5.  Re-contracting  with  the  organization  is 
needed  each  time  to  convince  the  members  and  managers  of  the  value  of  the  improved 
level.  At  higher  levels,  members  of  the  organization  can  see  the  values  of  earlier  prac¬ 
tices,  and  buy-in  deepens. 

•  At  the  early  levels  there  is  the  introduction  of  discipline;  at  the  higher  levels  the  organi¬ 
zation  must  add  and  refine  the  existing  discipline. 

•  There  is  a  shift  in  motivation  at  higher  levels.  The  organization  moves  from  escaping 
pain  to  seeking  improvement  opportunities. 

•  Process  maturity  does  not  always  equal  emotional  maturity.  There  are  defensiveness, 
posturing,  and  politics  at  all  levels. 

•  Humans  often  choose  the  familiar  and  comfortable  over  the  unknown. 

•  Understanding  and  commitment  are  deeper  at  the  higher  levels.  Compliance  and  rote 
works  at  lower  levels,  but  employees  need  internalization  and  understanding  for  higher 
levels. 

There  are  similarities: 

•  There  is  no  change  in  who  must  buy  in.  All  types  of  people  must  buy-in  at  each  level: 
the  staff  to  report  and  analyze  data  and  the  managers  to  use  the  data  properly. 

•  Development  of  and  conformance  to  process  is  an  evaluation  factor  for  everyone  in  the 
organization. 
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4.6  Working  Group  4:  Measurement  (Session  B) 

Participants:  Russ  Campbell,  Bill  Curtis,  Andre  Heijstek,  Pankaj  Jalote,  Keith  Joyce,  Walt 
Lipke,  Ray  Madachy,  Leitha  Purcell,  John  Smith,  Rakesh  Soni,  Prathima  Srinath, 
M.Thangarajan,  and  Dave  Zubrow. 

The  following  individuals  assumed  the  following  roles: 

•  Dave  Zubrow,  Morning  Facilitator 

•  Russ  Campbell,  Afternoon  Facilitator 

•  Leitha  Purcell,  Scribe 

•  M.  Thangarajan  (TH),  Timekeeper 

•  Ray  Madachy,  Presenter 


4.6.1  Observations 

The  group  began  by  selecting  a  focus  for  its  discussion.  The  initial  discussion  focused  on 
identifying  measurement  issues  associated  with  implementing  measurement  in  higher  matur¬ 
ity  organizations.  The  intent  was  to  identify  what  challenges  were  unique  to  measurement  in 
ML4  and  ML5  organizations.  While  some  unique  issues  were  identified,  much  of  the  list 
included  challenges  for  measurement  in  any  organization.  One  interesting  observation  is  that 
these  challenges  that  may  be  found  in  low  maturity  organizations  may  also  be  present  in 
higher  maturity  organizations. 

The  group  reached  consensus  on  pursuing  three  topics  in  depth: 

1 .  data  quality  and  cost  of  data  collection 

2.  capability  baselines  -  creation  and  use 

3.  tying  software  quality  to  business  objectives  and  identifying  the  relevant  process  measures 

4.6.2  Data  Quality  and  Cost  of  Data  Collection 

With  respect  to  the  first  topic,  it  was  noted  that  as  the  organization  matures  it  requires  data  of 
increasing  accuracy  and  granularity.  This  implies  data  should  be  recorded  as  close  to  real¬ 
time  as  is  feasible.  This  may  mean  that  effort  data  are  recorded  daily  and  other  data  are  re¬ 
corded  as  part  of  the  event  that  generates  them  (e.g.,  a  review,  test  or  inspection).  These  re¬ 
quirements  were  viewed  as  driving  up  the  costs  of  data  collection.  For  instance,  data  may 
need  to  be  collected  more  frequently  and  in  more  detail.  A  third  component  of  data  quality 
addressed  the  willingness  of  engineers  to  record  defects.  As  targets  are  set  for  internal  qual¬ 
ity  checkpoints,  this  may  motivate  some  underreporting  of  defects.  It  was  mentioned  that  it 
is  important  to  continually  communicate  the  use  of  the  data  being  recorded. 
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Finally,  the  group  seemed  to  support  the  statement  that  the  data  collection  done  in  support  of 
achieving  ML3  is  of  little  value  with  respect  to  the  use  and  needs  of  data  at  ML4.  The  ex¬ 
ception  to  this  that  was  noted  was  inspection  data.  However,  one  participant  during  review  of 
the  summary  felt  this  statement  was  too  strong.  Rather,  they  believe  that  the  data  collected  at 
ML3  may  not  be  sufficient  to  support  the  needs  of  ML4,  but  that  it  sets  the  stage  for  accom¬ 
plishing  ML4.  In  particular,  the  collection  of  data  around  Software  Product  Engineering  will 
help  establish  lifecycle  data  collection  and  analysis.  Similarly  the  measurements  around  In¬ 
tegrated  Software  Management  set  the  stage  for  continuous  improvement  at  ML5. 

4.6.3  Capability  Baselines  -  Creation  and  Use 

The  second  topic,  capability  baselines — creation  and  use,  is  one  more  uniquely  associated 
with  ML4  and  ML5.  Capability  baselines  are  established  as  a  result  of  analyzing  perform¬ 
ance  information.  For  baselines  to  be  useful,  they  need  to  be  based  on  the  experience  of 
similar  projects  or  process  enactments.  In  many  instances,  some  sort  of  subgrouping  of  the 
data  might  be  desirable  such  as  by  business  area  or  type  of  activity  (e.g.,  maintenance  vs.  de¬ 
velopment).  Additionally,  as  also  discussed  in  the  Statistics  working  group,  the  need  to  es¬ 
tablish  capability  baselines  based  on  nominal  process  executions  was  noted.  Not  all- 
available  data  will  be  appropriate  for  such  a  use. 

Of  some  concern  to  the  participants  was  the  seeming  conflict  between  developing  capability 
baselines  and  the  expected  culture  of  continual  improvement  at  higher  maturity  levels.  Par¬ 
ticipants  wondered,  if  the  process  is  continually  changing,  how  is  a  meaningful  capability 
baseline  to  be  established?  Indeed,  measuring  capability  as  distinct  from  measuring  im¬ 
provement  was  voiced  as  an  area  of  confusion.  During  review  of  the  draft  report,  it  was 
noted  that  this  distinction  is  important  and  the  following  explanation  and  example  was  of¬ 
fered.  “This  is  a  very  important  concept  in  understanding  the  difference  between  ML4  and 
ML3  and  ML5.  ML4  is  not  about  process  improvement;  it  is  about  process  management. 
There  is  a  close  analogy  here  to  the  hardware  world  were  calibration  equipment  is  used  to 
make  sure  the  hardware  is  operating  per  design.  Similarly  in  software,  ‘calibrated  processes’ 
such  as  an  inspection  process,  which  is  kept  stable  with  a  known  capability,  are  used  to  make 
sure  the  other  processes  such  as  requirements,  design,  and  integration  are  performing  as  de¬ 
signed.” 

Another  issue,  noted  as  somewhat  vexing,  was  what  the  heuristics  should  be  for  selecting 
processes  and  process  attributes  for  monitoring  and  control?  One  participant  noted  that  their 
organization  process  capability  baseline  was  composed  of  20  or  so  attributes.  While  projects 
would  only  control  6  -  12  of  those  attributes,  which  ones  would  be  controlled  was  not  well 
defined.  Guidance  based  on  the  nature  of  the  project  had  yet  to  be  defined.  One  example 
discussed  was  a  process  for  aligning  program  goals  and  customer  expectations  with  organ¬ 
izational  goals.  This  provided  insight  as  to  which  processes  should  be  placed  under  quantita¬ 
tive  control. 
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Another  issue  involved  making  comparisons  early  during  project.  It  was  noted  that  due  to  the 
paucity  of  available  data  during  the  early  days  of  a  project,  little  insight  was  gained  by  com¬ 
paring  the  data  with  a  capability  baseline. 

4.6.4  Tying  Software  Quality  to  Business  Objectives  and 
Identifying  the  Relevant  Process  Measures 

The  final  topic  discussed  involved  tying  software  quality  to  business  objectives  and  identify¬ 
ing  the  relevant  process  measures.  The  measures  may  address  process  efficiency,  effective¬ 
ness,  and  compliance.  It  was  generally  agreed  upon  by  the  participants  that  measures  need  to 
be  linked  to  business  drivers  and  that  the  characteristics  of  the  processes  will  constrain  the 
possible  set  of  measures  and  determine  their  operational  definitions.  A  particular  challenge  to 
doing  this  effectively  is  the  different  perspectives,  terminology,  and  motivations  that  need  to 
be  aligned.  Making  this  linkage  work  requires  vertically  traversing  many  layers  of  processes 
and  perspectives  within  the  organization.  And,  as  the  organization  increases  in  maturity,  the 
scope  of  involvement  within  the  organization  will  expand:  more  people  will  be  involved  in 
identifying  and  using  software  measures. 

4.6.5  Recommendations  for  High  Maturity  Organizations 

With  respect  to  the  first  topic,  data  cost  and  quality,  the  following  recommendations  were 
expressed. 

Do  Enforce  the  idea  that  staff  who  generate  data  should  get  to  use  it. 

Model  the  behavior  to  have  projects  use  data  by  having  measurement  group  work  with 
projects. 

Keep  the  linkage  to  use  and  goals. 

Keep  clear  whether  measures  are  for  the  enterprise  or  for  the  process  only. 

Have  statisticians  and  practitioners  collaborate  on  defect  and  effort  analysis. 

Define  analysis  at  same  time  as  defining  measures. 

Have  clear  process  definitions  and  define  measures  as  part  of  defining  processes. 

Have  management  pull  for  need  for  data. 

Have  common  measurement  criteria. 

Don’t  Fragment  collection  and  use  completely. 

Enter  the  same  data  multiple  times. 

Bite  off  more  than  you  can  chew. 

With  respect  to  the  second  topic,  capability  baselines  -  creation  and  use 

Do  Explore  data  to  understand  points  associated  with  special  causes  of  variation. 

Let  downstream  processes  (e.g.,  testing)  set  specs  for  upstream  processes  (e.g.,  defect  de¬ 
tection  activities  like  inspections). 
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Also  see  the  report  on  Statistics  for  related  observations  and  recommendations. 

With  respect  to  the  third  topic,  tying  software  quality  to  business  objectives  and  identifying 
the  relevant  process  measures 

Do  Define  process  requirements  like  you  would  a  product  requirement. 

Look  at  processes  that  are  involved  and  audience  to  identify  measures. 

Look  at  products  that  are  result  of  process  to  identify  entities  to  measure. 

Identify  critical  dimension  of  requirement  (cost,  schedule,  quality)  as  attributes  for  meas¬ 
urement. 


4.6.6  Recommendations  for  the  SEI 

No  specific  recommendations  were  made  for  the  SEI. 

4.7  Working  Group  5:  Technology  Transition 

Participants:  SuZ  Garcia,  Steve  Janiszewski,  L.  Ravichandran,  Prabhuu  Sinha,  and  Eileen 
Forrester. 

4.7.1  Goals  of  the  Working  Group: 

The  members  of  the  technology  working  group  framed  the  goals  as  a  set  of  questions  we 
would  like  to  be  able  to  answer: 

•  Are  the  issues  of  high  maturity  organizations  common  to  our  organizations? 

•  What  actions  can  we  recommend  for  common  problems? 

•  What  are  the  differences  between  our  issues  and  others? 

•  How  do  we  integrate  technology  deployment  with  process  deployment? 

•  Why  isn’t  technology  deployment  emphasized  earlier  in  CMMs? 

•  What  are  the  common  barriers  to  technology  deployment? 

-  Among  high  maturity  organizations? 

-  Between  high  and  low  maturity  organizations? 

•  When  are  different  technologies  optimally  introduced? 

Not  all  of  these  questions  can  be  adequately  addressed  in  a  one  day  working  group.  How¬ 
ever,  they  provided  the  framework  and  focus  for  the  discussions  that  occurred  throughout  the 
day. 

These  goals  translated  into  several  themes,  of  which  the  first  two  were  selected  for  particular 
focus: 
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•  processes  for  technology  adoption 

•  cultural  issues  in  facilitating  technology  adoption 

•  “systems  view”  of  technology  adoption 

•  tools  to  support  technology  adoption  (i.e.,  knowledge  management,  technology  adoption 
process  support) 

4.7.2  Approach 

To  address  the  goals  of  the  working  group,  two  approaches  were  agreed  upon: 

•  Enlarge  the  small  team  (4)  by  adding  a  “virtual”  team  -  the  25  participants  in  the  Sum¬ 
mer  1999  Workshop  on  Managing  Software  Innovation  and  Technology  Change,  repre¬ 
sented  by  Eileen  Forrester  of  the  SEI  [Forrester  99]. 

•  Use  a  couple  of  the  models  from  the  MSITC  Workshop  as  a  basis  for  framing  issues  and 
best  practices. 

The  two  models  used  from  the  workshop  include  the  Strategic  Planning  model  used  by  Lit¬ 
ton/PRC  and  the  INTRo  (Introducing  New  Technology  Rollout)  model  being  worked  on  by 
the  Accelerating  SW  Technology  Adoption  [Levine  99].  The  Litton-PRC  model  provides  a 
framework  for  understanding  how  explicit  technology  change  management  can  provide  a  link 
between  the  goals  of  the  business  and  the  performance  of  the  tasks  of  the  business.  The  IN¬ 
TRo  life  cycle  presents  a  set  of  phases  and  tasks  that  can  be  used  to  plan  and  manage  the  in¬ 
troduction  of  complex  technologies  into  one  or  more  organizational  units. 

Below  is  the  Unifying  Truism  from  the  MSITC  workshop: 

People's  ability  to  learn  and  absorb  change  is  the  new  constraint  on  the  speed  of 
technology  adoption. 

This  statement  from  the  Summer  99  MSITC  Workshop  resonated  strongly  with  the  experi¬ 
ences  of  the  technology  working  group  members.  It  emphasizes  the  importance  of  the  human 
issues  over  the  technical  issues  related  to  technology  adoption. 

4.7.3  INTRo  as  a  Basis  for  Identifying  Issues/Best  Practices 

The  major  phases  of  the  INTRo  life  cycle  are  diagrammed  below.  These  phases  were  used  to 
frame  the  answers  to  two  questions  that  all  members  of  the  working  group  worked  on  an¬ 
swering  for  their  own  organizations’  experiences: 

1 .  What  are  the  practices  that  most  contributed  to  their  success  in  performing  technology 
management  processes? 

2.  What  are  the  issues  that  still  provide  challenges  to  their  organizations,  despite  operating 
at  high  maturity  levels? 
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The  tables  following  the  diagram  provide  the  critical  success  factors  and  the  issues  that  were 
identified,  as  well  as  comment  that  arose  in  discussing  these  elements. 


Figure  1:  Major  Phases  of  INTRo  Life  Cycle 
Note  that  the  columns  in  the  tables  below  do  not  necessarily  represent  parallel  issues. 


Table  2:  Critical  Success  Factor  and  Issues  for  Project  Initiation 


;  Issues 

*  '  ■ 


Solution  is  often  pre-ordained. 


Limited  resources  mean  that  estab¬ 
lishing  ownership  for  a  proposed 
innovation  is  often  an  issue 

Too  busy — “If  it  ain’t  broke  don’t 
fix  it.” 


Critical  Success  Factors 

Practices  from  strategic  planning 
often  mature  ahead  of  or  in  parallel 
with  TCM  practices. 

Benchmark  against  relevant  com¬ 
panies  to  identify  potential  areas  of 
investment 

Track  emerging  technologies  with 
process  group  staff  or  other  full 
time  (non-project!)  staff. 


either  on  the  Issue 
(I)  or  the  Critical  Success 
Factor  (C) 

(I)  Not  that  common  unless  re¬ 
lated  to  strategic  issue. 


Table  3:  Organizational  Analysis 


Lack  of  shared  process,  model, 
etc  for  learning  and  communi¬ 
cating  about  cultural/social  issues 


Include  support  functions  and 
other  relevant  stakeholders 
early/at  this  stage. 


(I)  Less  prevalent  w/single-site 
or  single-discipline  organiza¬ 
tions 


Difficult  to  determine  when  the  Find  ways  to  bring  in  “social  sci-  (I)  Not  as  prevalent  with  single 

scope  of  impacts  has  been  cov-  ence”  experts  to  help.  site  organizations 

ered  (F)  Still  many  cultural  barriers  to 

social  scientists  participating 
with  engineering  organizations 
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Table  4:  Technology-Based  Solution  Definition 


Issues 


Difficult  to  dissociate  the  techni¬ 
cal  from  the  practical . 


Critical  Success  Factors 


“Proposer”  of  technology  adop¬ 
tion  follows  through  with  owner¬ 
ship  on  implementation. 


Notes  either  on  the  Issue 
(I)  or  the  Critical  Success 
|  Factor  (F) 

(I)  Solution  definition  vs.  tech¬ 
nology  selection. 

Also  a  practice  to  mitigate 
tendency  toward  lack  of  own¬ 
ership  in  project  initiation 


Market  constraints  that  lead  to 
internal  technology  development 
can  lead  to  large  long  term  costs. 


Table  5:  Technology  Selection 


Issues 


Generally  not  enough 
time/resources  to  scan  all  relevant 
alternatives. 


“Fire  Ready  Aim”  -  not  as 
prevalent  as  we  matured,  but  still 
a  “gotcha”  sometimes. 


Critical  Success  Factors 


“Technology  selection  board” 
including: 

-techies 

-business  planners 
-managers 

“Internal  competency  center”  fo¬ 
cusing  on  common  needs  for  sup¬ 
porting  technology  implementation 


Notes  either  on  the  Issue 
(I)  or  the  Critical  Success 
Factor  (F) 


Table  6:  Whole  Product  Definition/Development 


l  issues 


pQritical  Success  Factors 


V  f  'v  ' 


Notes  either  on  the  Issue 
(I)  or  the  Critical  Success 


Whole  product  not  perceived  as  the  Involve  “pilot”  groups  in  whole 
“real”  technology.  product  definition/development  to 

accelerate  buy-in. 


Often  difficult  to  “synchronize” 
involvement  of  the  relevant 
stakeholders. 


Train  teams  together  -  “boot  camp” 
-  plus  support  them  with 
mentoring. 


Whole  product  is  often  incom-  See  the  pilot  teams  as  the  “custom- 
plete  (it  isn’t  just  training!).  ers”  for  the  whole  product. 

Feedback  mechanisms  for  refin¬ 
ing  whole  product  often  missing. 
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Table  7:  Breakthrough 


Issues  Critical  Success  Factors  Notes  either  on  the  Issue 

(I)  or  the  Critical  Success 
Factor  (F) 

Getting  the  right  staff  at  the  right  Add  mentors/coaches  from  process 
time.  group  staffing/budget  to  add  incen¬ 

tive  to  pilot  teams  to  participate. 

Initial  pilot  works  but  rollout  Interview  pilot  candidates  to  de¬ 

steps  are  not  sufficiently  worked  termine  their  suitability  a  la  tech- 
out  (especially  in  project-focused  nology  adoption  curve, 
cultures). 

Finding  the  time  to  train  team  in  .  Be  sure  to  collect  before/after  data 
new  technology  prior  to  start  of  and  info, 
the  pilot. 

Difficult  to  find  credible  and 
relevant  pilots. 


Table  8:  Rollout 


Issues 


ss  Critical  Sue 


Critical  Success  Factors 

*  .  . 


•  J:  I 

■fWW|  Ww 

.  •  iV' 

■.  ' 

mm- 


. . ..  ...r „ ,.r.~  •- vs  -■ 

Notes  either  on  the  Issue 
(I)  or  the  Critical  Success 

Fact°MF) 


if? 


“Long”  (e.g.,  3  years)  life  cycle  of  “Sell”  the  breakthrough  and  relate 
projects  dilutes  ability  to  sue  pilot  it  to  the  organizational  objectives, 
results  in  rollout. 


Multiple  technology  adoptions  Advertise  the  objective  evidence, 
over  a  short  period  of  time  in¬ 
creases  confusion/resistance. 


How  to  move  from  successful  Provide  feedback  to  improve  the 

pilots  to  widespread  adoption  whole  products  so  organization 

with  project-focused  culture  un-  rollout  is  more  effective, 
clear. 

Plan  for  a  range  of  technology 
adoption  curve  populations  in  the 
pilots/rollouts. 
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4.7.4  Strategic  Planning  Insights 

The  diagram  below  is  an  adaptation  of  Litton-PRC’s  model  of  connecting  technology  man¬ 
agement  with  strategic  planning  (the  major  adaptations  were  to  orient  the  model  toward  a 
general  business  context  rather  than  toward  a  contract  business  model).  Below  the  diagram 
are  the  comments  made  by  working  group  members.  The  main  function  of  this  model  in  the 
working  group  was  to  provide  an  organized  framework  for  much  of  the  experience  that  they 
had  had. 


Obtain  business 


Technology  Change 
Management 


Contract  Execution 
or  Product  Development/Delivery 


Figure  2:  Strategic  Planning/TCM  Framework  adapted  from  Litton-PRC 


4.7.5  Strategic  Planning  Observations 

•  Both  product  technologies  and  infrastructure  technologies  contribute  to  strategic  planning 
and  obtaining  business. 

•  A  fundamental  issue  in  product  technology  management  is  whether  to  buy  companies 
who  already  provide  the  desired  product  technology  or  develop  internal  capability;  both 
have  advantages  and  disadvantages. 

•  Moving  strategic  planning  from  “closed”  to  “open”  in  terms  of  how  strategic  plans  are 
communicated  was  a  key  factor  cited  in  moving  technology  management  into  promi¬ 
nence  within  the  organization. 

•  For  organizations  that  are  geographically  distributed,  publication  and  communication  of 
the  strategic  plans  is  a  key  element  in  keeping  corporate  aligned  with  remote  sites  -  no 
matter  the  context.  This  is  a  continuing  challenge. 

•  A  continuing  issue  for  some  organizations  is  “What  is  the  right  reach?”  for  the  strategic 
plans?  One  high  maturity  organization  cited  a  move  from  a  plan  with  a  3-year  scope  and 
1-year  centers  to  a  plan  with  an  18-month  scope  and  6-month  centers  for  the  task  plan¬ 
ning  as  being  more  successful  for  them. 
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•  One  way  cited  to  obtain  sufficient  focus  on  technology  management  was  to  have  the  re¬ 
sponsibility  for  technology  change  management  be  seated  with  the  vice  president  of  En¬ 
gineering,  through  the  Process  Group  as  its  home. 

•  Research  and  development  of  product  line  technologies  as  well  as  infrastructure  tech¬ 
nologies  is  seen  as  necessary  scope  for  the  technology  change  management  processes. 

•  A  “line  of  sight”  from  strategic  plans  through  to  individual  objectives  was  necessary  to 
make  the  alignment  from  strategic  plans  and  business  goals  and  organizational  goals  a 
reality. 

•  At  least  one  annual  offsite  that  brings  together  strategic  business  planning  expertise  with 
technology  and  product  development  expertise  is  necessary  to  keep  the  technology  focus 
aligned  with  the  business  goals. 

4.7.6  TCM  Improvements  to  SW-CMM  vl.1 

The  working  group  primarily  has  experience  with  SW-CMM  vl.l,  so  this  was  the  base  used 
for  discussions  about  potential  improvements  to  the  TCM  key  process  area.  The  working 
group  recognizes  that  some  of  these  may  have  already  been  incorporated  into  SW-CMM  Ver¬ 
sion  2  and  the  CMM  Integration  Framework.  In  parentheses  after  some  comments  is  a  sug¬ 
gestion  of  what  element  of  the  CMM  might  be  the  best  place  to  address  this  comment.  The 
abbreviations  mean  the  following:  kp=key  practice,  sp=subpractice,  ab=ability  to  perform 
common  feature. 

•  Close  coupling  of  strategic  planning  processes  to  TCM  processes  requires  coverage,  (kp) 

•  Emphasize  “whole  product”  definition  and  development  as  part  of  TCM.  (kp) 

•  Couple  performance  objectives  to  technologies  planned  for  deployment  (e.g.,  as  whole 
product),  (sp) 

•  Add  coupling  of  proposing  and  implementing  as  strategy  to  get  ownership,  (ab) 

•  Bring  “pilot  teams”  into  planning  and  development  of  whole  products,  (kp) 

•  Address  cultural  implications  of  the  definition  of  the  scope  of  the  whole  product  (e.g., 
multi-site,  business  context). 

•  Include  direct  addressing  of  “impacted  areas”  in  technology/organizational  analysis. 

•  Add  more  differentiation  between  “incremental”  innovations”  and  “discontinuous  inno¬ 
vations.” 

4.7.7  Best  Practice  Recommendations  for  Successfully 
Deploying  Technologies 

At  the  end  of  the  working  session,  after  looking  at  all  the  material  generated,  the  working 
group  had  a  discussion  trying  to  summarize  the  most  important  things  their  organizations 
have  done  to  successfully  deploy  technology.  Some  of  these  are  repetitions  of  earlier  points; 
others  came  from  observing  what  we  had  done  to  answer  other  questions  and  from  seeking 
for  “what’s  missing.”  They  are  recorded  here  in  hopes  that  they  may  provide  useful  insights 
to  other  organizations  trying  to  improve  their  own  organizations’  ability  to  select  and  deploy 
worthwhile  technologies. 
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•  selecting  pilot  teams  based  on  technology  adoption  curve  (early  adopters  preferred) 

•  “line  of  sight”  coupling  of  technology  objectives  to  organization  and  individual  objec¬ 
tives,  and  budgeting  and  statusing 

•  separate  group  looking  externally  for  “good  matches”  -  NOT  on  project! 

•  widespread  publication/briefing  of  tactical  and  strategic  business  and  technology  plans 

•  making  the  proposer  the  owner  for  implementation 

•  establishing  technology  architecture  for  SPI  support  early;  deploying  supporting  tech¬ 
nologies  “just  in  time”  with  process  deployment 

•  “stop  the  world”  training  for  pilot  teams  and  explicit  coaching/mentoring 

-  mentor  does  NOT  equal  owner  of  the  adoption  (these  are,  however,  complementary) 

•  increased  sharing  of  technology  deployment  process  with  stakeholders  throughout  the 
TCM  process. .  .communicate. .  .communicate 

•  adding  mentors/coaches  from  process  group  and  budget  as  “extra”  staff  to  help  pilots  be 
successful 

•  establishing/employing  feedback/refinement  of  whole  product 

•  recognizing  that  first  iteration  does  NOT  equal  the  last  iteration 

•  identifying  the  owner  of  the  “sustaining”  part  of  technology  explicitly 

4.7.8  Recommendations  for  “TCM”  Material  Lower  in  SW-CMM 

One  of  the  more  provocative  discussions  within  the  working  group  was  how  technology 
adoption  is  and  is  not  addressed  within  the  SW-CMM.  SuZ  Garcia  provided  some  back¬ 
ground  on  the  placement  of  technology-related  material.  She  emphasized  that  the  placement 
of  technology-related  material  is  focused  on  where  in  the  organization’s  evolution  “mastery” 
of  technology  issues  is  expected  vs.  at  what  point  those  issues  would  begin  to  be  addressed. 
The  group  was  not  focused  on  moving  the  TCM  KPA  itself,  but  more  on  providing  support  in 
earlier  parts  of  the  model  for  providing  enabling  technology  support  for  software  practices, 
especially  for  software  process  improvement. 

Members  of  one  organization  asserted  that  an  emphasis  on  process-enabling  technologies 
from  the  beginning  of  their  SPI  effort  was  a  key  to  a  rapid  and  consistent  move  from  Level  1 
to  Level  5.  Their  analysis  of  the  SW-CMM  when  they  were  first  considering  it  as  their  im¬ 
provement  framework  was  that  carefully  deployed  technology  support  would  accelerate  pro¬ 
cess  discipline  buy-in.  They  have  coupled  their  technology  strategy  and  their  SPI  strategy 
from  the  beginning  of  their  SPI  effort  and  have  continued  with  this  path  throughout  their  ef¬ 
fort.  The  identification  of  a  technology  architecture  was  provided  with  their  initial  SPI  pro¬ 
posal  to  management,  and  the  technologies  were  planned  for  and  deployed  in  conjunction 
with  their  related  process  elements.  This  approach  was  different  from  the  other  high  maturity 
organizations  in  the  working  group;  however,  the  other  organizations  stated  that  they  might 
have  had  an  easier  time  at  some  points  in  their  improvement  efforts  had  they  considered  and 
employed  this  strategy. 
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The  primary  suggestions  provided  by  the  working  group  are 

•  OPF  should  address  integrating  process  architecture  and  technology  support  architecture 

•  Include  more  “feasibility”  focus  for  technology — encouraging  organizations  to  master 
analyzing  the  feasibility  of  different  technology  alternatives  earlier. 

-  Feasibility  focus  might  belong  either  as  part  of  OPF  or  in  the  Ability  to  Perform  sec¬ 
tion  of  different  KPAs 

-  Focus  on  ROI  for  software  process  improvement  within  OPF  -  “start  the  way  you 
mean  to  finish.” 

•  Add  subpractices/examples  on  appropriate  technology  support  in  relevant  KPAs  (e.g., 
problem  report  tracking,  defect  tracking). 

•  Add  more  front  matter  on  technology  implementation  vs.  TCM  mastery. 

•  Emphasize  appropriate  “data  collection”  technology  support  at  Level  2/3  and  above  as  a 
way  to  reduce  barriers  to  process  deployment — where  appropriate  technology  would  help 
with  process  adoption,  the  lack  of  encouragement  from  the  CMM  can  negatively  impact 
organizations  who  are  trying  to  follow  the  model  closely. 

4.7.9  Summary 

Technology  is  an  important  element  of  product  development  and  of  process  improvement. 
Selecting,  using,  and  deploying  it  appropriately  can  make  both  product  development  and  the 
improvement  of  product  development  easier  and  faster.  However,  making  appropriate  selec¬ 
tions  and  deploying  appropriately  are  the  keys  that  are  the  issues  of  both  low  and  high  matur¬ 
ity  organizations.  High  maturity  organizations  have  better  data  on  which  to  base  technology 
decisions.  However  they  still  have  challenges  in  appropriately  deploying  technology — the 
cultural  and  human  behavioral  issues  that  challenge  lower  maturity  organizations  still  show 
up  in  higher  maturity  organizations.  High  maturity  organizations  employ  differing  strategies 
to  address  these  issues;  however,  their  common  thread  is  tying  technology  use  to  business 
goals  and  involving  the  people  who  will  be  using  technology  in  planning  its  deployment. 


4.8  Working  Group  6:  Human  Issues 

Participants:  A1  Aldrich,  Julie  Barnard,  Kelley  Butler,  Harry  Carl,  Bhaskar  Chavali,  Ellen 
George,  Kelly  Gunning,  Barbara  Kolkhorst,  Judah  Mogilensky  (facilitator),  L.  Ravichandran, 
Sarala  Ravishankar,  Prabhuu  Sinha,  John  M.  Smith,  Agapi  Svolou,  Carolyn  Swanson,  M. 
Thangarajan,  Barbara  Tyson 

The  overall  topic  of  the  working  group  was  Human  Issues  in  High  Maturity  Organizations. 
The  general  thrust  of  the  group  was  to  discuss  these  ideas  and  to  record  perspectives,  but  not 
to  formally  endorse  any  specific  ideas  or  proposals  as  a  group. 
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The  group  began  by  collecting  potential  topics  to  discuss,  and  then  conducting  a  multi-vote  to 
identify  those  topics  that  the  group  was  most  interested  in  spending  time  on.  The  following  is 
the  list  of  potential  topics  that  was  collected;  the  highlighted  topics  were  the  ones  that  were 
chosen  for  discussion  time: 

1.  buy-in,  enthusiasm  about  process 

2.  cost,  schedule,  quality,  ownership  by  developers 

3.  worker  empowerment  in  process  definition  and  improvement 

4.  worker  empowerment  through  delegation  of  management  authority 

5.  encouraging  workers  to  report  accurate  data  (defects,  etc.) 

6.  sustaining  enthusiasm  in  a  growing  organization 

7.  difficulties  getting  new  staff  and  managers  up  to  speed 

8.  neutralizing  hot-shot  cowboy  developers 

9.  sustaining  enthusiasm  after  hitting  major  process  goals 

10.  how  cultures  (national,  organizational,  industry)  impact  process  discipline  and  im¬ 
provement 

1 1 .  growing  individual  contributors  in  a  team-oriented  environment 

12.  tension  between  more  process-oriented  and  less  process-oriented  projects  and  groups 

13.  how  to  motivate  data  collection  from  people  who  do  not  benefit  directly 

14.  observed  changes  or  evolution  in  interpersonal  behavior  patterns  as  organization  matures 

15.  avoiding  or  managing  dysfunctional  behavior  prompted  by  measurement 

16.  learning  to  build  team  chemistry 

17.  identification  and  development  of  core  competencies 

1 8.  keeping  people  on  large  projects  feeling  important  and  significant 

19.  similarities  and  differences  in  people  in  people  issues  between  lower  maturity  levels  and 
higher 

20.  skeptics — how  can  we  be  high  maturity  if  we  are  not  perfect 

The  following  sections  summarize  the  discussion  of  the  group  regarding  each  of  the  high¬ 
lighted  topics.  They  are  presented  in  the  order  in  which  they  were  discussed. 

4.8.1  Sustaining  Enthusiasm  after  Hitting  Major  Process 
Goals 

Issue:  How  to  sustain  enthusiasm  in  process  improvement  after  achieving  major  process 
goals.  “Continuous  improvement”  does  not  have  nearly  the  appeal  of  “getting  a  maturity 
level.”  (This  is  a  particular  issue  for  organizations  that  have  achieved  Level  5,  and  have  no 
more  maturity  level  goals  to  aim  for.) 
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Some  of  the  methods  offered  to  mitigate  this  issue  were 

•  internal  methods  (things  that  can  be  done  entirely  within  Level  4  and  5  organizations.) 

-  ISO  9000  registration  and  the  required  external  audits  every  six  months  help  main¬ 
tain  focus.  (Because  ISO  registration  requires  regular  re- validation,  unlike  CMM  as¬ 
sessment  results,  the  ISO  re-validation  exercises  provide  a  means  of  keeping  people 
in  the  organization  focused  on  process  issues.) 

-  ISO  9000-2000  will  place  increased  focus  on  Statistical  Process  Control  and  Con¬ 
tinuous  Improvement.  (This  new  focus  will  add  Level  4  and  Level  5  topics  to  the 
ISO  registration  and  re-validation  activities.) 

-  use  of  internal  awards  for  teams.  (Take  focus  away  from  improvement  as  measured 
by  maturity  levels,  and  put  more  emphasis  on  measuring  and  rewarding  business  re¬ 
sults,  now  that  the  organization  has  the  ability  to  do  such  measurement  with  reason¬ 
able  accuracy.) 

-  Set  a  pattern  of  ‘”set  a  goal,”  “achieve  goal’”  “set  a  new  goal” — eliminate  the  idea 
that  process  improvement  is  finished.  (Always  be  prepared  to  set  new  performance 
goals,  even  if  these  do  not  involve  new,  higher  maturity  levels.) 

•  external  methods  (things  that  other  organizations  can  do  to  help  Level  4  and  5  organiza¬ 
tions.) 

-  the  SEI  could  require  a  re-validation  of  Level  5  assessments  every  five  years.  (For 
the  first  time,  the  SEI  could  establish  renewal  requirements  for  CMM-based  maturity 
level  ratings.  Initially,  this  idea  would  be  applied  only  to  Level  5  organizations,  and 
only  at  five-year  intervals.  Later,  renewal  requirements  could  potentially  be  consid¬ 
ered  for  other  maturity  ratings.) 

-  DoD  requirements  for  SEI  CMM  Levels  and  assessment  recency.  (Many  DoD  pro¬ 
curements  that  require  submission  of  maturity  level  data  also  require  that  the  assess¬ 
ment  or  evaluation  not  be  more  than  so  many  months  or  years  old.  This  requirement 
can  prompt  organizations  to  conduct  re-assessments  after  a  period  of  time,  even 
though  there  no  expectation  of  a  higher  rating  than  the  Level  4  or  Level  5  achieved 
last  time.) 

-  requirements  for  more  public  sharing  of  ROI  data  and  benefits,  like  for  the  Baldrige 
and  the  IEEE  awards.  (Creating  a  culture  in  the  CMM  community  that  organizations 
achieving  Levels  4  and  5  are  expected  to  report,  at  least  once  but  perhaps  regularly, 
on  their  performance  and  on  the  measured  benefits  of  their  maturity  levels.) 

4.8.2  Encouraging  Workers  to  Report  Accurate  Data 

Issues 

•  In  some  cases,  workers  resist  data  reporting.  Why?  How  can  managers  overcome  this 

resistance? 

•  There  is  a  tradeoff  between  data  accuracy  and  data  cost.  How  can  managers  strike  the 

right  balance? 
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4.8.3  Resistance  to  Data  Reporting: 

Timekeeping:  Detailed  timekeeping  is  essential  to  quantitative  process  management.  How¬ 
ever,  many  managers  don’t  realize  how  much  of  their  time  (or,  how  little  of  their  time)  work¬ 
ers  spend  on  their  work  products.  This  time,  known  as  yield  or  duty  factor,  is  often  a  small  ' 
percentage  of  the  typical  workweek.  In  fact,  50%  yield  is  usually  considered  good,  and  60% 
is  considered  a  phenomenal  yield.  Lower-maturity  organizations  usually  don’t  collect  enough 
data  to  determine  yield,  so  workers  face  a  significant  issue  when  asked  to  start  detailed  time¬ 
keeping: 


“I  know  how  little  time  I  spend  on  my  work  products  and  planned  tasks  during  the  week,  and 
how  much  time  I  spend  in  meetings,  answering  management  questions,  and  taking  care  of  the 
unforeseen  (personal  and  work-related).  Management  has  no  clue  about  how  I  spend  my 
time.  What  happens  to  me  when  they  find  out?” 


There  are  at  least  two  keys  to  resolving  this  issue: 

•  Managers  need  to  get  past  their  surprise  at  the  timekeeping  data,  and  investigate  the  rea¬ 
sons  yield  is  lower  than  expected.  With  accurate  timekeeping  data,  managers  can  start 
fact-based  process  improvement.  This  improvement  works  on  two  levels:  first,  the  or¬ 
ganization  gets  a  better  process;  second,  workers  can  see  managers  using  the  data  for  im¬ 
provement,  not  unfair  retribution  against  workers.  This  helps  everyone  understand  how 
data  empowers  the  organization  to  improve. 

•  Managers  must  understand  that  being  human  is  a  legitimate  workplace  activity.  Workers 
need  to  talk,  and  not  always  about  work.  Workers  have  lives  outside  work,  and  there  are 
times  when  life  preempts  work.  Managers  must  let  their  people  know  that  these  human 
activities  are  an  acceptable  and  expected  part  of  life  in  the  workplace. 

Defect  Reporting:  There  is  a  natural  tendency  for  workers  to  avoid  reporting  their  own  de¬ 
fects.  Here  are  some  key  points: 

•  Managers  need  to  show  workers — by  example — how  to  use  defect  data  to  improve  proc¬ 
esses. 

•  Managers  must  avoid  using  defect  data  for  individual  retribution,  so  fear  is  driven  out. 

(A  point  that  was  often  stressed  by  W.  Edwards  Demming.) 

•  Defining  “defect”  is  critical.  Some  organizations  don’t  consider  unit  test  errors  to  be  de¬ 
fects,  but  others  do.  Some  organizations  don’t  count  defects  until  peer  review  of  unit  test 
results.  PSP  counts  compile  errors  as  defects.  There  is  no  clear  consensus,  and  managers 
need  to  carefully  evaluate  the  tradeoffs  based  on  their  business  needs  and  software  proc¬ 
esses. 

•  Some  organizations  use  independent  testers  from  unit  test  through  final  delivery  to  help 
ensure  accurate  defect  reporting. 

•  Some  organizations  make  reported  defect  quantity  a  key  metric  for  each  individual, 
though  they  also  include  other  measurements  to  help  ensure  the  defect  reporting  isn’t  in¬ 
flated.  Control  charts  of  defect  data  also  highlight  under-reporting  of  defects. 
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•  Defect  data  collected  by  organizations  represented  in  the  group  included  defect  density  at 
delivery  and  “leak-through,”  i.e.,  defects  injected  in  one  phase,  but  detected  during  a 
later  phase. 

4.8.4  Balancing  Data  Accuracy  and  Data  Cost 

The  discussion  then  returned  to  the  timekeeping  topic.  Accurate  timekeeping  data  is  essential. 
However,  most  organizations  find  it  easier  to  collect  effort  data  for  process  management 
separate  from  their  accounting  and  payroll  systems.  The  duplicate  data  entry  costs  less  than 
the  effort  and  time  required  to  get  required  process  data  from  the  accounting  and  payroll  sys¬ 
tems.  Accounting  and  payroll  systems  focus  on  data  required  to  pay  employees  and  bill  cus¬ 
tomers,  not  data  required  to  manage  processes.  It  is  usually  more  effective  to  use  a  separate 
system  for  recording  total  time  spent  on  all  activities,  so  the  time  can  be  categorized  by  proj¬ 
ect,  phase,  activity,  or  other  required  category. 

If  task  effort  is  different  from  paid  effort — as  is  often  the  case  when  workers  are  salaried — 
there  can  be  complications  from  US  Labor  Department  definitions  of  hourly  and  salaried  em¬ 
ployees,  and  rules  governing  their  treatment. 

Most  organizations  automate  data  collection  and  analysis  because  it  helps  maintain  accuracy, 
reduces  delays,  and  improves  efficiency.  However,  it  is  important  to  maintain  support  staff 
for  the  timekeeping  and  data  collection  systems.  Cutting  support  staff  to  reduce  costs  doesn’t 
make  sense  unless  data  collection  is  also  reduced,  thereby  reducing  process  effectiveness. 
Someone  must  still  collect  data,  and  if  there  is  no  support  staff,  then  higher-paid  professional 
staff  must  do  the  work.  (Shifting  work  from  lower-paid  support  staff  to  higher-paid  profes¬ 
sional  staff  does  not  seem  to  be  a  way  to  achieve  cost  savings,  but  organizations  seem  to  try  it 
all  the  time.) 

There  is  a  natural  link  between  task  effort,  task  results,  and  earned  value.  If  effort  data  is  col¬ 
lected  separately  from  accounting  and  payroll  data,  the  effort  data  can  drive  the  earned  value 
system.  The  accounting  and  payroll  data  generally  cannot  be  used  for  earned  value,  because 
those  data  do  not  reflect  the  total  task  effort,  e.g.,  if  technical  people  are  salaried  or  work  un¬ 
compensated  time. 

4.8.5  Notes  Regarding  PSP  Implementation 

One  of  the  organizations  represented  in  the  Human  Issues  group  had  implemented  PSP  over  a 
year  ago.  Here  are  some  relevant  points  from  that  experience: 

•  Managers  were  very  surprised  by  initial  yields  averaging  9  hours  per  week  (out  of  a 
nominal  40  work  hours  per  week,  only  9  were  actually  spent  doing  real  project  work). 
Once  managers  attacked  the  underlying  process  problems,  yield  improved  until  it  reached 
15  hours  per  week  one  year  later. 
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•  The  organization  developed  a  tool  for  individuals  to  track  time  very  precisely.  When  a 
worker  starts  a  planned  task,  the  worker  starts  a  timer  on  the  workstation  screen.  When 
work  stops  for  any  reason,  the  worker  stops  the  timer.  (Operation  of  the  tool  is  similar  to 
a  “chess  clock.”) 

•  The  task  timer  tool  made  the  impact  of  interruptions  obvious,  and  created  demand  for 
better  time  management.  Since  management  was  considered  a  primary  source  of  inter¬ 
ruptions,  management  designated  each  morning  for  “quiet  time:”  no  phone  calls  returned, 
no  meetings,  and  no  email  read  or  answered.  This  contributed  significantly  to  improved 
yield.  (In  a  similar  vein,  some  organizations  have  tried  designating  one  day  per  week  as  a 
“meeting-free  day,”  again  to  increase  time  on  task.) 


4.8.6  Avoiding  or  Managing  Dysfunctional  Behavior  Prompted 
by  Measurement 


Issue:  The  act  of  measurement  can  engender  dysfunctional  behavior,  i.e.,  focus  on  what  is 
measured  to  the  neglect  of  non-measured  duties  of  equal  importance/criticality.  (This  argu¬ 
ment  is  made  very  effectively  in  the  book  Measuring  and  Manaeing  Performance  in  Organi¬ 
zations  by  Robert  D.  Austin,  Dorset  House  Publishing,  1996.) 

Some  of  the  ways  to  mitigate  this  issue  are 

•  Be  aware/sensitive  the  impact  of  data  can  have  on  individuals  (see  issue  statement 

above). 

-  Understand  that  being  human  and  doing  human  activities  is  legitimate,  i.e.,  human 
activities  must  be  allowed. 

-  Understand  that  an  “on-task”  percent  of  work  time  of  50%  is  VERY  good;  most  of 
the  remainder  is  taken  up  with  management  required  tasks;  only  a  little  is  taken  up 
with  “being  human”  tasks. 

•  Don’t  use  process  data/metrics  results  to  reward  or  punish  individuals. 

-  According  to  Deming,  teams,  not  individuals,  should  be  rewarded.  For  example,  one 
organization’s  members  described  their  reward  breakdown  as  follows:  30%  based  on 
achieving  individual  goals,  40%  based  on  team  goals,  and  30%  based  on  the  organi¬ 
zation’s  goals. 

-  Understand  and  differentiate  between  measures  used  to  reward  a  team  for  project 
execution  vs.  measures  used  as  a  basis  for  hiring,  firing,  promotion  of  individuals 

-  Take  the  time  to  make  what  some  people  view  as  “scary”  measurements  as  “okay” 
within  the  organization,  i.e.,  acceptable  and  normal  in  the  organization’s  culture. 

Plan  for  the  fact  that  some  measurements  will  take  a  “long”  time  to  establish. 

-  Use  data  to  remove  people  from  a  project,  NOT  from  the  organization.  This  move¬ 
ment,  in  cases  where  unsuccessful  managers  were  returned  to  engineering  duties, 
worked  out  well  for  the  individuals,  i.e.,  they  were  happier  after  the  change. 

•  Use  mentoring  and  coaching  to  discuss  where  obstacles  are  and  determine  what  help  in¬ 
dividuals/teams  need  to  succeed. 

-  One  approach  used  is  to  perform  a  skills  review  at  different  project  milestones  in¬ 
stead  of  one  time  only.  (That  is,  at  each  milestone,  review  the  skills  required  for  the 
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next  pieces  of  work  vs.  skills  present  on  the  team,  and  take  action  as  needed.  This  is 
contrasted  with  performing  a  skills  review  once,  at  the  start  of  a  project,  and  then 
never  re-visiting  it.) 

•  Many  program  managers  (PMs)  don’t  really  understand  what  it  takes  to  develop  soft¬ 
ware.  You  must  help  the  PMs  and  other  managers  gain  a  more  holistic  view  of  software 
development.  They  must  understand  what  skills  and  knowledge  are  needed  for  each  indi¬ 
vidual  to  perform. 

•  Change  manager’s  role  (e.g.,  the  SEPG’s  role)  to  include  the  creation  of  a  culture  that 
accepts  metrics  and  their  use  as  acceptable  and  beneficial. 


Example  1  of  what  worked  for  one  organization  is  described  below: 

•  The  organization  had  evolved  a  culture  where  is  was  not  “okay”  to  ask  for  help,  so  junior 
people  spent  a  long  time  struggling  with  problems  even  though  mentors  were  available  to 
help  them.  To  address  this  problem,  everyone  was  given  a  red,  yellow,  green  balloon. 
When  a  problem  arose  the  person  lowered  the  green  balloon  and  raised  the  yellow.  If  the 
person  could  not  solve  the  problem  in  a  specified  reasonable  time,  the  yellow  was  re¬ 
placed  with  a  red  balloon.  Mentors  were  always  available  to  identify  and  work  problems, 
and  they  would  watch  the  balloons,  so  they  would  know  where  to  go.  This  approach 
worked  well  in  a  “bay”  like  environment  where  the  balloons  can  be  easily  seen  over  cu¬ 
bicle  walls.  The  approach  was  clearly  acceptable  in  this  organization,  and  it  created  an 
“okay”  way  to  ask  for  help. 


Example  2  of  what  worked  follows: 

•  Organization  established  a  firm  policy  of  NOT  accounting  for  non-task  hours,  and  only 
tracking  on-task  activities,  and  allowing  for  the  addition  of  non-planned  tasks.  The  con¬ 
cern  was  to  avoid  the  impression  of  a  “big  brother  watching.” 

•  People  were  given  the  task  of  developing  their  own  task  plan  for  improving  selected 
measures.  Based  on  their  own  recorded  data,  they  determined  if  they  had  met  their  task 
plan,  and  they  would  take  corrective  action  or  not,  as  warranted.  This  was  all  done  pri¬ 
vately  by  individuals,  so  there  was  no  management  imposed  visibility,  yet  normal  com¬ 
petitive  peer  pressure  can’t  be  hidden  so  everyone  strives  to  do  better. 

An  alternative  view  of  the  strategy  employed  in  Example  2,  as  implemented  by  another  or¬ 
ganization:  Time  tracking  codes  were  established  for  some  non-tasks  hours,  e.g.  meetings, 
briefing,  training,  etc.  This  organization  felt  that  tracking  measures  for  some  non-task  hours 
was  both  acceptable  to  the  workers  and  provided  useful  data. 


4.8.7  Tension  Between  More  Process-Oriented  And  Less 
Process-Oriented  Groups  And  Projects 


Issue:  While  most  projects  in  high  maturity  organizations  demonstrate  a  very  high  commit¬ 
ment  to  processes,  there  may  be  a  few  projects  that  do  not  display  the  same  enthusiasm  for 
processes.  This  “backward”  behavior  by  one  or  two  projects  leads  others  to  question  why 
they  are  working  so  hard  to  follow  process.  Similarly,  the  software  group  may  be  following 
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processes,  but  the  other  groups  like  hardware,  etc.,  may  not  have  rigorous  processes.  This 
again  leads  to  questions  like  “Why  just  us,  but  not  hardware,  others  ?”  This  invariably  causes 
tension  between  more  process-oriented  and  less  process-oriented  groups  and  projects. 

During  the  group  discussions,  it  emerged  that  this  issue  is  typically  faced  when  an  organiza¬ 
tion  is  experiencing  a  large  growth,  because  of  customer  resistance  to  process  adherence,  or 
because  of  rapid  technology  change. 

•  Large  or  rapid  growth  leads  to  major  assimilation  challenges,  resulting  in  varying  degrees 
of  process  institutionalization.  As  process  internalization  and  ownership  is  not  uniform 
across  the  organization,  it  becomes  easier  for  small  projects  to  become  the  first  ones  to 
adhere  to  processes,  and  for  larger  projects  to  lag  behind.  In  such  situations,  the  problem 
tends  to  go  away  with  time,  implying  that  the  issue  is  a  characteristic  of  rollout,  but  not 
of  later  phases. 

•  Some  projects  may  become  laggards  because  of  customer  resistance  to  process,  as  the 
customers  communicate  that  “Level  3  is  all  I  need,  I  won’t  pay  for  more  than  that”  or 
“ISO  is  a  resource  drain.”  In  such  cases,  customer  education  is  important,  and  organiza¬ 
tions  have  to  convey  to  the  customer  that  it  will  cost  the  customer  more  to  do  it  “the 
customer’s  way,”  which  has  had  the  desired  result.  (This  is  an  instance  of  the  process 
improvement  theme  of  “putting  customers  in  touch  with  the  consequences  of  their  own 
actions.”) 

•  When  new  technology  is  adopted,  the  projects  have  a  tendency  to  believe  that  the  “old” 
process  does  not  apply  to  their  project.  (Depending  on  the  situation,  there  may  be  some 
legitimacy  to  this  claim.  However,  that  becomes  an  argument  to  update  the  process,  not 
abandon  it.) 


Because  of  the  varying  nature  of  individuals  in  an  organization,  there  will  be  pockets  of  lag¬ 
gards  who  will  resist  any  change.  It  appears  that  many  laggards  do  catch  up  over  time,  and 
the  issue  goes  away.  However,  there  are  some  laggards  who  never  catch  up,  who  then  be¬ 
come  “second  class  citizens”  and  impede  the  work  of  the  rest  of  the  organization.  It  now 
becomes  a  management  challenge  to  deal  with  such  situations  effectively.  Managers  and 
customers  need  to  have  the  right  data  to  take  correct  decisions. 

At  the  end  of  the  discussion,  the  myth  regarding  the  “conflict”  between  creativity  and  disci¬ 
pline  was  explored.  Many  staff  believe  that  processes  tend  to  stifle  creativity.  The  example 
of  artists  who  spend  a  long  time  learning  the  discipline  of  their  art  before  becoming  creative 
in  their  fields  was  discussed,  i.e.,  it  is  the  discipline  that  enables  and  leads  to  creativity. 

4.8.8  Similarities  and  Differences  in  People  Issues  between 
Lower  and  Higher  Maturity  Levels 

Issue:  What  are  the  major  similarities  in  the  people  issues  between  lower  and  higher  maturity 
levels,  that  is,  what  issues  tend  to  persist  without  significant  change  as  the  organization  pro¬ 
gresses?  And,  what  are  the  key  differences  in  the  people  issues  between  lower  and  higher 
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maturity  levels,  that  is,  what  issues  tend  to  go  away  at  higher  levels,  or  only  emerge  at  higher 
levels? 

It  was  the  consensus  of  the  group,  after  discussion,  that  there  are  behavioral  clues  that  are 
indicative  of  the  maturity  level  of  the  organization.  That  is,  organizations  of  different  matur¬ 
ity  levels  behave  differently.  The  following  ideas  were  developed: 

There  are  differences: 

•  Buy-in  is  a  continuous  process.  But  buying  in  to  CMM  SW  Levels  2  and  3  does  not 
automatically  lead  to  buy  in  for  Levels  4  and  5.  Re-contracting  with  the  organization  is 
needed  each  time  to  convince  the  members  and  managers  of  the  value  of  the  improved 
level.  At  higher  levels,  members  of  the  organization  can  see  the  values  of  earlier  prac¬ 
tices  and  buy-in  tends  to  deepen. 

•  At  the  early  levels,  there  is  the  introduction  of  discipline  where  perhaps  none  had  existed 
before;  at  the  higher  levels  the  organization  must  add  and  refine  the  existing  discipline. 

•  There  is  a  shift  in  motivation  at  higher  levels.  The  organization  moves  from  escaping 
pain  (e.g.,  failed  projects,  lost  contracts,  lost  customers,  etc.)  to  seeking  improvement  op¬ 
portunities.  (This  is  a  challenge  to  management,  driving  change  from  a  basis  of  seeing 
how  things  could  be  better,  even  though  they  are  okay  now,  as  opposed  to  driving  change 
from  a  basis  of  avoiding  visible  and  obvious  bad  outcomes.) 

•  Understanding  and  commitment  are  deeper  at  the  higher  levels.  Compliance  and  rote 
following  of  processes  that  are  not  well  understood  can  work  at  lower  levels,  but  em¬ 
ployees  need  internalization  and  understanding  for  higher  levels  to  be  achieved. 


There  are  similarities: 

•  There  is  no  change  in  who  must  buy  in.  All  types  of  people  must  buy  in  at  each  level:  the 
staff  to  report  and  analyze  data  and  the  managers  to  use  the  data  properly. 

•  Process  maturity  does  not  always  equal  emotional  maturity.  There  are  defensiveness, 
posturing,  and  politics  at  all  levels. 

•  Humans  often  choose  the  familiar  and  comfortable  over  the  unknown.  (This  can  be  true 
at  all  levels.) 

•  Development  of  and  conformance  to  process  is  an  evaluation  factor  for  everyone  in  the 
organization. 

4.9  Working  Group  8:  Software  Quality  Assurance 

Participants:  Dottie  Acton,  Colin  Benton,  Rick  Biehl,  Russ  Campbell,  Joe  Puffer,  Prathima 

Srinath,  Warren  Schwomeyer 
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4.9.1  Discussion  Framework 

This  paper  presents  the  results  of  the  working  group  on  Software  Quality  Assurance.  The 
working  group  looked  at  the  multiple  ways  in  which  the  SQA  role  changes  as  an  organization 
matures,  and  on  the  impact  of  those  changes  on  the  organization  and  on  the  individuals 
within  the  SQA  group. 

The  discussion  of  the  working  group  was  framed  around  a  two  dimensional  framework;  one 
dimension  being  the  five  successive  maturity  levels  of  the  CMM,  and  the  second  dimension 
being  the  topics  of  inquiry  being  discussed  by  the  group.  These  topics  included 

•  Independence  to  Objectivity  -  The  extent  to  which  the  independence  of  the  SQA  group 
within  the  organizational  structure  influences  the  objectivity — real  or  perceived — they 
bring  to  bear  on  their  activities. 

•  Organizational  Resistance  -  The  extent  to  which  the  organization  resists  the  role  and 
function  of  SQA  as  the  processes  of  the  organization  mature. 

•  Customer  Integration  -  The  extent  to  which  the  customer  can  be  integrated  as  a  natural 
partner  in  the  SQA  process. 

•  Push  to  Pull  -  The  extent  to  which  the  SQA  dynamic  shifts  through  maturity  from  the 
SQA  organization  pushing  itself  onto  the  project  arena,  to  the  project  environment  pull¬ 
ing  the  SQA  function  in. 

•  Assurance  Target  -  The  extent  to  which  the  target  of  SQA  activities  shifts  from  simple 
standards  compliance  toward  more  effective  improvement  strategies  as  maturity  pro¬ 
gresses. 

•  Skill  Requirements  -  The  extent  to  which  the  skills  required  of  the  SQA  function  change 
as  maturity  levels  are  progressed. 

•  Roles  and  Responsibilities  -  The  extent  to  which  the  expectations  for  roles  and  respon¬ 
sibilities  associated  with  SQA  shift  as  maturity  progresses. 

•  People  Impact  -  The  extent  to  which  SQA  affects  the  career  paths  and  opportunities  of 
its  practitioners  in  differing  ways  as  maturity  progresses. 

Discussion  progressed  from  topic  to  topic,  identifying  and  noting  the  characteristics  and  be¬ 
haviors  of  the  discussion  area  as  it  progresses  through  the  maturity  model.  This  provided  a 
vehicle  for  understanding  the  SQA  transition  from  each  distinct  perspective. 


4.9.2  Observations 

Results  of  each  of  the  specific  discussion  topics  are  detailed  below. 


4.9.2.1  Independence  to  Objectivity 

Summary:  In  a  Level  1  organization,  personal  contact  by  SQA  provides  visibility  into  the 
project  activities.  Since  this  is  the  case,  there  is  a  tendency  to  equate  independence  with  ob¬ 
jectivity,  since  the  separation  gives  the  freedom  to  make  less  than  popular  decisions.  At 
Level  2,  the  project's  documented  processes  provide  a  framework  for  the  SQA  visibility.  At 
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Level  3,  SQA  is  more  integrated  with  the  teams  and  can  begin  to  use  data  to  provide  visibility 
into  the  project  status.  At  Level  4,  the  detailed  data  that  is  available  provides  the  objectivity 
needed,  even  without  organizational  independence.  At  Level  5,  the  process  itself  provides 
inherent  objectivity,  and  SQA  can  step  back  and  observe  at  a  higher  level. 

The  CMM  suggests  that  the  SQA  group  has  an  independent  reporting  channel  to  senior  man¬ 
agement  and  that  this  independence  should  result  in  confidence  that  objective  information  is 
being  reported. 

There  are  two  elements  to  be  considered  in  tracking  the  maturing  of  the  implementation  of 
the  SQA  function  from  Level  1  to  Level  5.  Visibility  of  the  SQA  group  into  the  process  and 
product  quality  is  the  first  element.  This  relates  to  how  information  upon  which  assessments 
are  based  is  obtained.  The  second  element  is  Objectivity.  This  element  relates  to  how  the 
reporting  objectivity  is  ensured. 

At  Level  1,  the  SQA  function  may  not  even  be  performed.  When  it  is  performed,  visibility  is 
obtained  through  personal  contact.  SQA  representatives  are  present  at  all  inspections,  at  pre¬ 
scribed  levels  of  testing,  at  all  status  meetings,  etc.  SQA  obtains  the  needed  information  by 
being  present  when  it  is  being  generated  and  observing  it  first  hand.  At  this  level,  the  organ¬ 
izational  independence  is  equated  with  objectivity.  The  SQA  group  reports  through  an  inde¬ 
pendent  channel,  therefore  its  reporting  “must”  be  objective.  This  independence  also  pro¬ 
duces  an  SQA  group  that  is  separate  from  the  development  group.  An  “us  vs.  them” 
perspective  often  develops  between  the  two  groups,  which  enhances  the  perceived  objectiv¬ 
ity. 

The  project’s  defined  processes  provide  visibility  at  Level  2.  Each  project’s  processes  are 
defined  and  documented  in  advance.  This  allows  the  SQA  group  to  “see”  what  is  going  to 
happen  on  the  project  and  develop  an  SQA  Plan  that  begins  to  select  the  areas  for  SQA  over¬ 
sight.  The  project  planning  performed  at  Level  2  provides  objective  expectations  for  the 
whole  team,  including  SQA.  The  defined  processes  and  project  plans  establish  a  phased  ap¬ 
proach  to  the  project  presenting  SQA  with  differing  opportunities  for  assessment  and  review 
in  each  phase.  At  this  level,  SQA  is  starting  to  be  viewed  as  a  member  of  the  team. 

SQA  becomes  a  full  member  of  the  team  with  the  establishment  of  Integrated  Product  Teams 
at  Level  3.  Improved  data  collection,  analysis  and  retention  (e.g.,  central  repository)  provide 
improved  visibility  and  objectivity  at  Level  3.  The  granularity  of  information  available  in¬ 
creases  the  opportunities  for  a  wider  array  of  assurance  activity  and  involvement  with  the 
project.  The  data  begins  to  “speak  for  itself.” 

Level  4  performance  provides  more  detailed  data  and  more  mature  analysis  techniques.  Ob¬ 
jectivity  results  from  listening  to  what  the  data  says  about  process  and  product  quality. 
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Self-initiated,  continuously  improving  processes  contain  inherent  objectivity  at  Level  5.  At 
this  level,  SQA  can  step  back  from  the  process  and  observe.  Visibility  is  achieved  through 
sampling  and  selective  involvement.  Observation  of  perturbations  in  a  well-running  process 
provides  SQA’s  impetus  for  action. 

SQA  changes  along  with  the  software  development  process  in  a  maturing  organization.  The 
means  by  which  SQA  obtains  visibility  into  the  process  and  product  quality  transitions  from 
being  additional  activities  to  being  integrated  activities  to  being  observational  activities  with 
negligible  disruption  to  the  basic  software  development  and  management  activity.  The  un¬ 
derstanding  of  objectivity  changes,  allowing  a  maturing  organization  more  flexibility  in  im¬ 
plementing  the  SQA  function.  The  organization  relies  less  on  organizational  independence 
and  more  on  data  objectivity. 

4.9.2.2  Organizational  Resistance 

Summary:  In  a  Level  1  organization,  there  is  overt  resistance  to  SQA,  both  because  of  the 
perception  that  SQA  costs  too  much,  and  the  confrontational  nature  of  the  police  activities 
that  SQA  is  forced  to  perform.  At  Level  2,  and  3  this  abates  somewhat  with  the  cultural 
change  that  accepts  the  SQA  role  as  part  of  the  process;  however  covert  resistance  is  some¬ 
times  present.  It  isn't  until  Level  4  that  most  projects  will  have  the  detailed  data  that  can  al¬ 
low  them  to  understand  the  value  of  QA.  At  Level  5,  with  the  focus  on  improving  the  proc¬ 
ess,  the  relationship  with  SQA  becomes  one  of  collaboration  and  encouragement  rather  than 
confrontation. 

At  Levels  1  and  2,  SQA  is  forced  on  projects  that  typically  do  a  poor  job  of  estimating.  Con¬ 
sequently,  SQA  exacerbates  their  cost  and  schedule  overruns.  Customers  are  unwilling  to  pay 
for  SQA  because,  in  their  view,  the  developers  should  take  care  of  quality  as  a  matter  of  engi¬ 
neering  excellence.  Projects  are  also  unwilling  to  pay  for  SQA  which  provides  little  in  the 
way  of  useful  service  or  feedback. 

Rules  get  in  the  way  of  progress  and  entrepreneurial  project  management,  resulting  in  project 
confrontation.  Finger  pointing  occurs  along  with  “in  your  face’  audits.  The  evaluators  who 
typically  have  poor  software  engineering  skills  cause  friction  when  finding  product  errors. 
Typical  defects  found  by  SQA  are  superficial,  cause  lots  of  paperwork,  and  aren’t  always  re¬ 
ported  fairly.  SQA  is  not  usually  a  partner  before  Level  3.  Overt  resistance  is  typical,  with 
an  “us  vs.  them”  attitude  prevailing.  SQA  is  not  invited  to  many  activities. 

A  culture  change  occurs  on  the  way  to  Level  3.  There  is  an  acceptance  of  SQA’s  role,  and  it 
becomes  part  of  the  formal  development  process.  As  IPTs  form,  SQA  is  included  from  the 
beginning.  The  instantiation  of  peer  reviews  along  with  product  diversity  causes  SQA  to  take 
on  a  process  auditing  role.  SQA  begins  to  use  the  science  of  sampling  to  assure  coverage  with 
a  small  staff.  However,  the  organization  is  increasingly  heterogeneous  with  projects  at  all 
levels  of  maturity.  The  SQA  function  must  interface  with  these  projects,  and  provide  the 
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proper  support.  Friction  and  organizational  resistance  occur  with  the  lower  maturity  level 
projects,  and  when  a  Level  1  auditor  works  on  a  project  exhibiting  Level  3  maturity. 

It  is  only  as  the  organization  implements  Level  4  practices,  that  there  is  enough  data  to  under¬ 
stand  the  value  of  SQA.  Organizational  resistance  disappears  as  the  data  captures  the  cost  of 
quality,  and  the  value  proposition  for  having  SQA  functions  is  articulated.  SQA  loses  the  pro¬ 
cess  and  product  police  stigma.  They  are  watching  quality  through  organizational  data,  which 
removes  them  from  confrontational  situations.  SQA  actively  works  defect  prevention  issues 
along  with  the  developers,  which  reinforces  the  idea  of  partnership.  While  heterogeneity  of 
service  increases,  the  SQA  management  is  able  to  use  data  to  provide  appropriate  service  at 
all  maturity  levels  without  invoking  resistance. 

The  organization  works  with  the  QA  function  in  a  collaborative  manner  at  Level  5.  SQA  is 
encouraged  to  assist  with  process  management  and  technology  problems  encountered  on  the 
projects.  The  SQA  mentoring  function  at  Level  5  overcomes  any  resistance  that  might  be  left. 

4.9.2.3  Customer  Integration 

Summary:  In  a  Level  1  organization,  SQA  is  often  performing  in  the  role  of  the  eyes  and  ears 
of  the  customer,  with  a  focus  on  ensuring  that  customers  get  the  product  that  they  expect.  At 
Level  2,  SQA  begins  to  monitor  the  state  of  the  process  as  well  as  the  product,  still  serving  as 
the  eyes  and  ears  of  the  customer,  but  with  earlier  opportunities  for  feedback.  At  Level  3, 
usually  a  change  occurs  with  the  introduction  of  IPTs  that  include  the  customer.  At  this  point, 
there  is  less  value  in  the  SQA  role  as  a  customer  surrogate  since  the  customer  is  involved  in 
the  process. 

At  Level  1,  SQA  serves  as  the  eyes  and  ears  of  the  customer.  The  customer  is  primarily  in¬ 
terested  in  the  product  being  produced,  and  has  to  be  strongly  encouraged  to  show  interest  in 
the  process  being  used  by  the  project.  SQA  often  provides  the  only  visibility  the  customer 
can  get  to  product  quality.  As  maturity  progresses  toward  Level  2,  the  customer  begins  to 
care  more  about  the  process,  but  still  doesn’t  typically  become  a  participating  stakeholder  in 
the  process. 

By  Level  3,  customers  are  more  involved  in  the  process;  often  through  participation  in  IPTs. 
Customers  are  no  longer  outside  observers  to  the  process;  rather  they  are  participating  in  pro¬ 
cess  activities  directly  and  are  often  taking  responsibility  for  their  own  actions  in  the  plan. 
SQA  is  still  participating,  but  tends  to  take  more  objective  positions  on  the  team  as  customers 
take  responsibility  for  their  own  interests. 

As  projects  approach  Level  4  maturity,  customers  have  gained  enough  confidence  in  the  pro¬ 
cess  to  begin  taking  greater  risks,  including  participating  in  process-based  incentive  plans  and 
increased  customer  project  ownership.  By  Level  5,  customers  see  the  process  as  an  opportu¬ 
nity  for  mutual  improvement  and  the  sharing  of  opportunities  identified  by  any  stakeholder. 
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The  line  between  customer  and  provider  is  not  clearly  evident  in  the  process,  reducing  the 
need  for  SQA  to  serve  as  customer  surrogate. 

4.9.2.4  Push  to  Pull 

Summary:  In  a  Level  1  or  2  organization,  SQA  is  frequently  only  present  on  a  project  be¬ 
cause  it  is  required,  either  by  the  customer  or  by  organization  policy.  At  the  higher  levels, 
SQA  is  usually  present  on  a  program  from  the  beginning,  as  an  integral  part  of  the  program 
team. 

At  Level  1  the  organization  lacks  management  practices  and  has  different  objectives.  Occa¬ 
sionally,  managers  impose  a  systematic  approach  to  software  development  and  maintenance, 
but  resort  to  shortcuts  during  crisis.  In  some  cases,  ad  hoc  practices  may  be  enforced.  They 
are  normally  reactive  measures.  Generally,  at  this  level,  quality  assurance  is  customer  focused 
and  largely  pushed  onto  projects. 

During  institutionalization  of  management  practices,  defined  practices  are  implemented  by 
projects  because  the  products  and  activities  are  reviewed,  audited  and  reported.  They  include 
SQA  because  it  is  required.  As  maturity  improves,  organizational  push  for  disciplined  ap¬ 
proach  is  much  more  evident  than  customer  push.  The  organization  as  a  whole  sets  the  poli¬ 
cies  to  implement  management  practices  and  ensure  compliance. 

At  higher  maturity  levels,  service  is  context  dependent.  Customer  participation  in  product  and 
process  reviews  is  planned.  The  SQA  group  and  customer  representative  are  part  of  project 
and  are  involved  with  the  project  from  initiation  and  planning  stage  to  establishment  of  plans, 
standards,  and  procedures.  Activities  are  performed  according  to  the  plan.  SQA  provides 
visibility  on  process  compliance,  and  sees  itself  further  pulled  into  the  process. 

At  Level  4,  quantitative  measures  are  adopted  to  track  and  control  the  project  and  the  process. 
This  empowers  each  project  to  manage  based  on  actual  data.  Management  and  control  is 
driven  by  organizational  assets.  By  Level  5,  SQA  has  been  pulled  completely  into  the  proc¬ 
ess  as  an  integral  part  from  the  beginning.  The  systematic  approach  is  inherent  in  software 
development  and  maintenance  activities.  It  is  no  longer  customer  driven  or  organization 
driven.  Quality  assurance  is  built  into  project  requirement. 

4.9.2.5  Assurance  Target 

Summary:  At  Level  1,  the  SQA  focus  is  on  standards  compliance,  usually  after  the  fact.  In 
essence,  SQA  is  asking  “did  you  produce  the  correct  product?”  At  Level  2,  it  becomes  possi¬ 
ble  for  SQA  to  verify  in-process  compliance,  and  to  ask  “are  you  following  the  steps  needed 
to  produce  a  quality  product?”  At  Level  3,  this  in-process  compliance  activity  increases,  es¬ 
pecially  with  the  introduction  of  inspections.  At  Level  4,  SQA  is  able  to  challenge  the  or¬ 
ganization  by  asking  if  the  metrics  show  “can  you  get  there?”  As  the  project  teams  begin  to 
control  their  processes  to  manageable  targets,  SQA  can  also  ask  “will  you  get  there?”  At 
Level  5,  SQA  becomes  an  integral  part  of  the  process  and  product  improvement  activities. 
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At  Level  1,  SQA  focuses  on  standards  compliance,  usually  after  the  fact  on  projects.  In  many 
Level  1  organizations,  a  general  absence  of  standards  results  in  SQA  activities  centered 
around  industry  norms  and  conventions.  SQA  is  typically  asking  if  the  projects  have  done 
what  was  expected;  a  strictly  reactive  mode. 

By  Level  2,  enough  process  has  been  defined  for  SQA  to  begin  actions  based  on  in-process 
compliance.  SQA  impact  can  be  immediate,  but  is  still  largely  reactive  because  of  a  con¬ 
tinuing  lack  of  visibility  into  the  detail  processes.  As  visibility  increases  at  Level  3,  SQA  can 
have  more  immediate  impact;  challenging  projects  as  they  progress.  The  introduction  of 
more  systematic  reviews  and  inspections  shifts  more  of  the  SQA  focus  to  the  project  teams. 
The  shift  of  SQA  staff  from  reactivity  toward  future  planning  takes  place  during  this  period. 

By  Level  4,  challenge  has  turned  to  control;  as  SQA  becomes  able  to  monitor  manageable 
targets  and  the  projects  take  on  more  and  more  of  the  SQA  responsibility  embedded  in  their 
plans.  SQA  focuses  on  future  planning  with  projects;  assuring  that  projects  will  be  able  to 
work  toward  their  own  quality  goals.  By  Level  5,  SQA  focus  is  exclusively  on  process  and 
product  improvement,  with  project  appraisal  and  conformance  issues  completely  shifted  to 
the  project  teams. 

4.9.2.6  Skill  Requirements 

Summary  :  At  Level  1,  members  of  the  SQA  organization  need  to  be  detail-oriented,  with 
strong  audit  skills.  At  Level  2,  the  skill  requirements  increase  to  include  basic  metrics  skills 
and  software  skills.  With  the  progression  to  Level  3,  there  is  a  need  for  facilitation  skills, 
process  improvement  skills,  and  more  advanced  metrics  and  analysis  skills.  At  Level  4,  con¬ 
sulting  skills  are  needed,  as  is  increased  depth  and  breadth  in  software  skills.  Level  5  adds 
more,  including  defect  prevention  and  causal  analysis,  mentoring  and  advanced  modeling 
skills. 

At  Level  1,  members  of  SQA  need  to  be  very  detail-oriented;  with  strong  audit  and  coordina¬ 
tion  skills  as  they  deal  with  almost  exclusively  low  maturity  project  behaviors.  As  maturity 
levels  increase,  SQA  staff  will  need  to  take  on  the  knowledge  implied  by  the  improvement 
process  areas;  often  leading  such  learning  in  the  organization,  while  offering  support  to  others 
working  on  similar  learning  curves. 

As  the  organization  progresses  to  Level  2,  SQA  people  must  improve  their  basic  metric  skills; 
as  well  as  overall  software  project  and  engineering  knowledge.  Their  coordination  skills 
must  expand  into  facilitation;  being  able  to  inform  and  guide  projects  toward  process  im¬ 
provement. 

By  Level  3,  SQA  staff  must  have  the  more  advanced  metrics  and  analysis  skills  needed  to 
help  the  organization  and  projects  benefit  from  early  metrics.  Such  successes  are  key  factors 
in  building  support  for  continuing  the  improvement  initiatives.  Facilitation  skills  expand 
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here  into  a  broader  consulting  skill  set  as  SQA  staff  work  with  more  people  in  the  organiza¬ 
tion. 

As  the  focus  of  improvement  becomes  more  targeted  toward  Level  4,  SQA  staff  must  main¬ 
tain  their  knowledge  base  in  software  and  software  engineering  in  order  to  be  able  to  partici¬ 
pate  in  the  discussions  that  will  surround  early  defect-analysis  activities.  That  knowledge, 
supported  by  skills  in  defect  and  causal  analysis  will  make  the  SQA  a  valued  resource  to 
these  higher  maturity  projects. 

By  Level  5,  SQA  staff  require  the  ability  to  mentor  others  on  the  staff,  guiding  them  along  the 
same  learning  path  that  the  SQA  staff  member  has  followed.  In  addition  to  analysis  skills, 
the  SQA  member  must  develop  skills  in  general  modeling  as  higher  maturity  improved  met¬ 
rics  become  available  requiring  such  analysis.  SQA  staff  will  need  to  maintain  all  of  these 
levels  of  skill  simultaneously  in  order  to  continue  to  support  the  broad  range  of  projects  en¬ 
countered  in  such  high  maturity  organizations. 

4.9.2.7  Roles  and  Responsibilities 

Summary:  At  Level  1,  SQA  responsibilities  are  frequently  limited  to  reactive,  after-the-fact 
reporting.  At  Level  2,  this  changes  to  a  stop-the-line  mentality,  as  in-process  problems  are 
detected.  SQA  performs  its  activities  through  attendance  at  meetings  and  through  direct 
oversight  of  program  and  subcontractor  activities.  At  Level  3,  they  are  more  involved  with 
project  activities  such  as  inspections,  and  can  provide  more  current  reporting  of  project  suc¬ 
cess  or  issues.  At  Level  4,  a  significant  change  takes  place  since  they  can  now  use  sampling 
to  obtain  data,  and  can  provide  before-the-event  projective  reporting.  At  Level  5,  this  can 
progress  to  being  able  to  provide  an  organizational  as  well  as  project  viewpoint  to  manage¬ 
ment. 

At  Level  1  the  role  of  SQA  is  one  of  policing  the  software  development  project.  SQA  is  re¬ 
sponsible  for  reporting  typically  to  the  project  customer  how  well  the  project  has  met  con¬ 
tractual  requirements.  Reporting  is  after  the  fact  which  leads  to  a  reactive,  disjointed  rela¬ 
tionship  between  the  project  and  SQA.  The  focus  of  SQA  is  on  project  completeness. 

As  a  project  moves  to  Level  2,  SQA  continues  in  the  policing  role.  Members  of  SQA  typi¬ 
cally  attend  reviews,  inspections,  and  status  meetings.  Reporting  is  still  typically  after  the 
fact  with  a  “stop  the  line”  mentality.  SQA  findings  are  written  and  dispositioned  before  the 
project  can  complete. 

SQA  in  a  Level  3  organization  takes  on  a  more  proactive  role  of  directly  monitoring  the  proj¬ 
ect  activities  and  work  products.  Reporting  of  process  execution  and  product  quality  occurs 
concurrently  with  the  development.  The  focus  is  on  project  success. 

As  an  organization  moves  to  Level  4  the  role  of  SQA  becomes  one  of  indirectly  monitoring 
the  project  activities  and  products.  The  engineering  group  accomplishes  many  quality  assur- 
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an ce  activities.  SQA  monitors  the  accomplishment  of  activities  such  as  inspections  and  re¬ 
views.  Sampling  of  process  results  and  product  quality  is  performed. 

An  organization  operating  at  Level  5  will  be  implementing  processes  that  watch  themselves 
through  the  continuous  collection  and  analysis  of  data.  The  SQA  role  is  one  of  projecting  the 
project  outcome  based  on  the  processes,  technologies,  and  resources  in  place.  Reports  are 
provided  to  management  and  the  team  members  before  events  occur  to  ensure  a  successful 
project.  The  focus  is  on  organization  success. 

4.9.2.8  People  Impact 

Summary:  At  Level  1,  SQA  is  seen  as  a  limited  career  path,  for  individuals  with  fewer 
skills.  As  the  organization  matures,  they  must  gain  the  vision  of  where  the  organization  is 
going,  and  begin  to  look  ahead  to  the  next  steps  in  the  process  for  the  project.  By  the  time 
the  organization  and  SQA  reach  Level  5  maturity,  SQA  is  seen  as  a  leader — the  choice  career 
path  for  the  best  the  organization  has  to  offer. 

SQA  is  seen  as  a  very  limited  career  path  at  Level  1 .  It  is  often  presumed  to  be  a  side-track  as¬ 
signment  or  individuals  with  fewer  skills.  Many  SQA  people  in  low  maturity  organizations  see 
themselves  as  being  "punished"  in  some  way;  often  after  having  excelled  as  project  leaders. 

As  the  organization  matures  through  Levels  2  and  3,  SQA  people  must  gain  the  vision  of 
where  the  organization  is  going,  and  begin  to  look  ahead  to  the  next  steps  in  the  process  for 
the  project.  They  must  become  active  participants  in  the  planning  and  management  side  of 
process  maturity.  They  can't  spend  all  of  their  time  in  trenches  with  the  projects  unless  they 
desire  to  rotate  back  to  direct  project  assignments  shortly.  As  a  path  into  management,  SQA 
requires  a  broader  participation  and  visibility  across  the  entire  organization. 

As  the  organization  and  SQA  progress  through  Level  4  to  Level  5,  SQA  is  seen  as  a  leader  in 
process  improvement  specifically,  and  the  organization  generally.  SQA  represents  a  viable 
and  preferred  career  path  for  the  best  the  organization  has  to  offer. 
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5  Considerations  for  Next  Steps 


The  High  Maturity  Workshop  was  favorably  reviewed  by  the  participants,  as  was  the  survey 
of  high  maturity  organizations.  In  the  workshop  evaluation,  56%  of  the  participants  sug¬ 
gested  the  workshop  should  be  an  annual  event,  and  the  rest  preferred  an  18-24  month  cycle. 
As  part  of  the  SEI’s  ongoing  communications  with  this  segment  of  the  Software  CMM  user 
community,  it  is  our  intent  to  make  both  the  workshop  and  the  survey  regular  events,  occur¬ 
ring  on  a  roughly  annual  cycle. 

Although  the  high  maturity  segment  of  the  software  community  is  growing  rapidly,  there  is 
some  question  about  what  “high  maturity”  really  means.  The  Software  CMM  does  not  have 
as  complete  a  picture  of  Levels  4  and  5  as  is  captured  for  Levels  2  and  3,  thus  the  opportunity 
for  high  maturity  organizations  to  discuss  directly  the  issues  and  challenges  they  have  to  deal 
with  provides  a  valuable  dialog  for  both  the  companies  and  the  SEI. 

There  is  reason  to  believe  that  some  organizations  have  taken  an  overly  liberal  interpretation 
of  Levels  4  and  5  in  the  Software  CMM.  This  is  similar  to  the  situation  in  1990,  when  sig¬ 
nificant  consistency  and  reliability  issues  with  Level  2  and  3  assessments  were  reported.  This 
was  a  particular  concern  for  organizations  pursuing  government  contracts  where  the  results  of 
software  capability  evaluations  against  the  SEI  maturity  framework  of  that  time  were  a  factor 
in  source  selection.  The  most  significant  step  in  addressing  this  problem  was  the  publication 
of  Software  CMM  v  1.0  in  1991,  which  provided  a  comprehensive  description  of  Levels  2  and 
3. 

The  current  release  of  the  Software  CMM,  Version  1.1,  was  released  in  1993.  A  conservative 
stance  was  taken  in  defining  Maturity  Levels  4  and  5  because  of  the  sparsity  of  Level  4  and  5 
organizations.  We  have  learned  much  about  high  maturity  practices  since  then,  through 
mechanisms  such  as  the  high  maturity  workshops,  but  the  fundamental  principles  of  Levels  4 
and  5  are  not  as  clearly  articulated  in  Version  1.1  as  we  might  wish.  The  planned  release  of 
Software  CMM  v2  in  1997  was  halted  in  favor  of  work  on  CMM  Integration.  Drafts  of  the 
CMMI  model  are  available  at  http://www.sei.cmu.edu/cmm/cmms/cmms.integration.html  and 
incorporate  much  of  the  current  thinking  on  Level  4  and  5  practices,  but  the  operational 
model  today  remains  Software  CMM  vl.l  as  released  seven  years  ago. 

The  SEI  is  taking  steps  via  papers,  training,  and  other  mechanisms  to  address  this  problem. 
SEI  training  on  high  maturity  practices  and  statistical  process  control  for  software  is  now 
available,  but  it  will  take  time  to  deploy  these  courses  to  the  software  process  improvement 
community  and  the  Lead  Assessors.  Work  continues  on  understanding  high  maturity  prac- 


CMU/SEI-2000-SR-003 


87 


tices  better  through  the  workshops  and  surveys,  and  it  is  hoped  that  better  communication 
between  the  high  maturity  organizations  and  Lead  Assessors  will  clarify  many  of  the  high 
maturity  challenges  that  the  software  community  faces. 

The  issues  surrounding  quantitative  management — including  both  measurement  and  statisti¬ 
cal  process  control — are  challenges  even  for  high  maturity  organizations.  Even  the  growing 
use  of  control  charts  as  a  standard  part  of  the  software  process  should  not  suggest  that  they 
become  a  requirement  for  high  maturity.  There  are  still  unanswered  questions  regarding  the 
business  value  of  various  quantitative  management  techniques  in  different  business  environ¬ 
ments  and  in  conjunction  with  different  software  engineering  methodologies.  For  example, 
most  high  maturity  organizations  today  are  in  high  reliability  environments,  building  real¬ 
time  applications  and/or  embedded  systems  [Paulk  00].  Will  control  charts  provide  signifi¬ 
cant  business  value  in  a  commercial  shrink-wrap  environment?  Conceptually,  Levels  4  and  5 
of  the  Software  CMM  are  based  on  stable  and  capable  processes  as  traditionally  controlled  by 
SPC.  Empirically,  we  are  still  exploring  the  implications  of  these  concepts  in  a  design¬ 
intensive,  human-centered  process. 
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Appendix  A:  List  of  Workshop  Participants 


Dorothy  Acton,  Lockheed  Martin  Mission  Systems,  Gaithersburg,  MD 

Armistead  Aldrich,  Lockheed  Martin  Mission  Systems,  Gaithersburg,  MD 

Mark  Amaya,  SPP&T,  El  Toro,  CA 

Paul  Andersen,  NCR  Corporation,  San  Diego,  CA 

Julie  Barnard,  United  Space  Alliance,  Houston,  TX 

Colin  Benton,  Alenia  Marconi  Systems,  Cambradley,  United  Kingdom 

Richard  Biehl,  Data-Oriented  Quality  Solutions,  Orlando,  FL 

Kelley  Butler,  USAF,  Tinker  AFB,  OK 

Russell  Campbell,  Lockheed  Martin  Tactical  Aircraft  Systems,  Ft.  Worth,  TX 

Harry  Carl,  Honeywell,  Minneapolis,  MN 

Jeffery  Carter,  NCR  Corporation,  San  Diego,  CA 

Ranjan  Chak,  Oracle  Software  India  Limited,  Bangalore,  India 

Bhaskar  Chavali,  NIIT,  New  Delhi,  India 

Mary  Beth  Chrissis,  The  Software  Engineering  Institute,  Pittsburgh,  PA 
Bill  Curtis,  TeraQuest  Metrics,  Inc.,  Austin,  TX 
Donna  Dunaway,  The  Software  Engineering  Institute,  Pittsburgh,  PA 
Kenneth  Dymond,  Process  Transition  International,  Inc.,  Annapolis,  MD 
Suzanne  Garcia,  aimware  Inc.,  Pittsburgh,  PA 
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Ellen  George,  AlliedSignal  Aerospace  Co.,  Teterboro,  NJ 

Kelly  Gunning,  Marconi  Integrated  Systems,  San  Diego,  CA 

Rick  Hefner,  TRW  Inc.,  Redondo  Beach,  CA 

Andre  Heijstek,  Ericsson,  PN  Gouda,  Netherlands 

Barbara  Hirsh,  Motorola  Inc.,  Woodridge,  IL 

Watts  Humphrey,  The  Software  Engineering  Institute,  Pittsburgh,  PA 

Pankaj  Jalote,  I.I.T.  Kanpuri, ,  India 

Stephen  Janiszewski,  AlliedSignal  Aerospace  Co.,  Teterboro,  NJ 

Keith  Joyce,  Marconi  Integrated  Systems,  San  Diego,  CA 

Bijay  Kumar  Jyotishi,  Wipro  Technologies,  Bangalore,  India 

Barbara  Kolkhorst,  IBM  Global  Services,  College  Station,  TX 

Anand  Kumar,  CitiCorp  Information  Technology  Industries,  Ltd.,  Mumbai,  India 

A  Kumaran,  Oracle  Software  India  Limited,  Bangalore,  India 

Walter  Lipke,  OC  ALC,  Tinker  AFB,  OK 

Ray  Madachy,  Litton  Guidance  and  Control  Systems,  Woodland  Hills,  CA 
Steve  Masters,  CISE,  Pittsburgh,  PA 
Andrew  Meadow,  PRC,  Inc.,  McLean,  VA 

Judah  Mogilensky,  Process  Enhancement  Partners,  Inc.,  Silver  Spring,  MD 
Jane  Moon,  Raytheon,  Fullerton,  CA 
Linda  Morris,  NCR,  San  Diego,  CA 

Mark  Paulk,  The  Software  Engineering  Institute,  Pittsburgh,  PA 
Mary  Lynn  Penn,  Lockheed  Martin,  Philadelphia,  PA 
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William  Peterson,  The  Software  Engineering  Institute,  Pittsburgh,  PA 
Neil  Potter,  The  Process  Group,  Dallas,  TX 
Joseph  Puffer,  TeraQuest  Metrics,  Inc.,  Austin,  TX 
Leitha  Purcell,  Northrop  Grumman  Corporation,  Pico  Rivera,  CA 
Lakshminarayanan  Ravichandran,  HCL  Perot  Systems,  Noida,  India 
Sarala  Ravishankar,  Motorola  India  Electronics  Ltd.,  Bangalore,  India 
Warren  Schwomeyer,  Lockheed  Martin  Federal  Systems,  Owego,  NY 
Michael  Scott,  Raytheon,  Tucson,  AZ 

Joseph  Seppy,  Software  Productivity  Consortium,  Herndon,  VA 

Prabhat  Kumar  Sinha,  Satyam  Computer  Services  Limited,  Hyderabad,  India 

John  Smith,  Noumena  Consulting  Group,  Inc.,  Beavercreek,  OH 

Rakesh  Soni,  HCL  Perot  Systems,  Noida,  India 

Phillip  Sperling,  Telos  Federal  Systems,  Lawton,  OK 

Prathima  Srinath,  IBM  Global  Services,  North  Plainfield,  NJ 

Agapi  Svolou,  Agapi  Svolou,  Independent  Consultant,  Pittsburgh,  PA 

Carolyn  Swanson,  Consultant,  Plano,  TX 

M.  Thangarajan,  Tata  Elxsi  Limited,  Bangalore,  India 

Barbara  Tyson,  The  Software  Engineering  Institute,  Arlington,  VA 

Subramanyam  Venkata,  Wipro  Technologies,  Bangalore,  India 

Ramesh  Venkatraman,  BFL  Software  Limited,  Bangalore,  India 

Ramaswami  Viswanathan,  Wipro  GE  Medical  Systems,  Bangalore,  India 

Charles  Weber,  The  Software  Engineering  Institute,  Boulder,  CO 
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Donald  White,  Lockheed  Martin  Undersea  Systems,  Manassas,  VA 

Gary  Wigle,  The  Boeing  Company,  Seattle,  WA 

David  Zubrow,  The  Software  Engineering  Institute,  Pittsburgh,  PA 
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Appendix  B:  List  of  High  Maturity 
Organizations  Participating 


The  following  list  of  high  maturity  organizations  lists  most  of  the  known  Level  4  and  5  or¬ 
ganizations.  The  organizations  that  participated  in  the  workshop  are  noted  with  a  V  in  col¬ 
umn  1  of  the  table.  A  more  comprehensive  list,  including  points  of  contact,  dates  of  assess¬ 
ment,  and  Lead  Assessors  is  maintained  at  <URL:  http://www.sei.cmu.edu/cmm/ 
cmm.articles.html  #high-mat-orgs>. 

As  of  February  15,  2000,  the  full  list,  of  which  the  published  list  is  a  subset,  includes 

■  44  Level  4  organizations 

■  27  Level  5  organizations 

■  26  non-US  high  maturity  organizations 

-  1  ML4  in  Australia 

-  14  ML4  in  India 

-  10  ML5  in  India 

-  1  ML4  in  Israel 

26  high  maturity  organizations  participated  in  the  workshop.  Of  the  26,  there  were  represen¬ 
tatives  from  12  companies  in  India. 

Please  be  aware  of  the  following  issues  regarding  this  list. 

•  The  SEI  does  not  certify  companies  at  maturity  levels. 

•  The  SEI  does  not  confirm  the  accuracy  of  the  maturity  levels  reported  by  the  Lead  Asses¬ 
sors  or  organizations. 

•  This  list  of  Level  4  and  5  organizations  is  by  no  means  exhaustive;  we  know  of  other 
high  maturity  organizations  that  have  chosen  not  to  be  listed. 

•  The  SEI  did  not  use  information  stored  within  its  Process  Appraisal  Information  System 
to  produce  this  document. 

•  The  organizations  listed  gave  explicit  permission  to  publish  this  information. 

•  No  information  obtained  in  confidence  was  used  to  produce  this  list. 


Workshop 

Participant 

High  Maturity  Organization 

V 

BFL  Software  Limited,  Bangalore,  India 

The  Boeing  Company,  Aircraft  &  Missiles  &  Phantom  Works  Southern  Califor¬ 
nia,  Long  Beach,  CA 
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Workshop 

Participant 

High  Maturity  Organization 

The  Boeing  Company,  Military  Aircraft  &  Missile  Systems  F/A-18  Mission 
Computer,  St.  Louis,  MO 

The  Boeing  Company,  Reusable  Space  Systems  and  Satellite  Programs, 

Huntington  Beach  &  Seal  Beach,  CA 

V 

The  Boeing  Company,  Space  Transportation  Systems,  Kent,  WA  [Fowler  97, 

Wigle  97,  Yamamura  97] 

CG-Smith  Software,  Bangalore,  India 

V 

Citicorp  Information  Technology  Industries  Limited  (CITIL),  Mumbai,  India 

Cognizant  Technology  Solutions,  Chennai,  India 

DSQ  Software,  Chennai,  India 

Future  Software  Private  Limited,  Chennai,  India 

V 

HCL  Perot  Systems,  Noida  and  Bangalore,  India 

Honeywell  International,  Avionics  Integrated  Systems  (formerly  AlliedSignal, 
Guidance  &  Control  Systems),  Teterboro,  NJ 

V 

IBM  Global  Services  India,  Bangalore,  India 

International  Computers  India  Ltd.  (ICIL),  Pune,  India 

V 

Litton  Guidance  and  Control  Systems,  Woodland  Hills,  CA 

V 

Lockheed  Martin  Federal  Systems,  Owego,  NY 

V 

Lockheed  Martin  Management  &  Data  Systems,  King  of  Prussia,  PA 

V 

Lockheed  Martin  Mission  Systems,  Gaithersburg,  MD 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems  -  Syracuse,  Syra¬ 
cuse,  NY 

Lockheed  Martin  Naval  Electronics  &  Surveillance 

Systems  -  Eagan,  Eagan,  MN 

V 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems  -  Manassas  (for¬ 
merly  Undersea  Systems),  Manassas,  VA 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems  -  Moorestown, 
Moorestown,  NJ 

Lockheed  Martin  Space  Electronics  and  Communications  Systems  -  Manassas 
(formerly  Loral  Federal  Systems),  Manassas,  VA 

Motorola  Australia  Software  Centre,  Adelaide,  Australia 

V 

Motorola,  GSM  (Global  System  for  Mobile  Communications)  Systems  Division, 
Network  Systems  Group,  Arlington  Heights,  IL  [Miller  98] 

V 

Motorola  India  Electronics  Ltd.  (MIEL),  Bangalore,  India  [Ravishankar  99] 

NCR  Corporation,  Teradata  Development  Division,  Massively  Parallel  Systems, 
San  Diego,  CA 

V 

Northrop  Grumman,  Air  Combat  Systems,  Integrated  Systems  and  Aeronautics 
Sector,  El  Segundo,  CA 

Northrop  Grumman,  Integrated  Systems  &  Aerostructures,  AEW  &  EW  Systems 
(formerly  Surveillance  &  Battle  Management),  Bethpage,  NY 

V 

Oracle  Software  India  Limited,  India  Development  Center,  Bangalore,  India 
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Participant 

High  Maturity  Organization 

Raytheon  (formerly  Raytheon  E-Systems),  Garland,  TX 

Raytheon  C3I  Fullerton  Integrated  Systems,  Command  and  Control  Sys¬ 
tems/Middle  East  Operations,  Fullerton,  CA 

Raytheon  Missile  Systems,  Software  Engineering  Center,  Tucson,  AZ 


J  Satyam  Computer  Services  Ltd.,  India 


Tata  Consultancy  Services,  HP  Centre,  Chennai,  India 

Tata  Consultancy  Services,  SEEPZ,  Mumbai,  India 

Tata  Consultancy  Services,  Shollinganallur,  Chennai,  India 

Tata  Consultancy  Services,  US  West,  Chennai,  India 

V 

Tata  Elxsi  Limited,  Bangalore,  India 

Telcordia  Technologies,  Piscataway,  NJ  [Ferrara  00] 

V 

United  Space  Alliance,  Space  Shuttle  Onboard  Software  Project,  Houston,  TX 
[Billings  94,  Fishman  97,  Krasner  94,  Paulk  95] 

US  Air  Force,  Ogden  Air  Logistics  Center,  Technology  &  Industrial  Support 
Directorate,  Software  Engineering  Division,  Hill  AFB,  UT  [Cosgriff  99a,  Cos- 
griff  99b,  Craig  99,  Oldham  99,  Paulk  99] 

US  Air  Force,  Oklahoma  City  Air  Logistics  Center,  Directorate  of  Aircraft  Man¬ 
agement,  Software  Division,  Test  Software  and  Industrial  Automation  Branches 
(OC-ALC/LAS),  Tinker  AFB,  OK  [Butler  95,  Butler  97] _ 

US  Army,  Communications  and  Electronics  Command  (CECOM),  (Software 
Engineering  Center  (SEC),  Fire  Support  Software  Engineering,  Fort  Sill,  OK 

US  Navy,  Fleet  Material  Support  Office,  Mechanicsburg,  PA 


V 

Wipro  GE  Medical  Systems,  Bangalore,  India 

V 

Wipro  Technologies,  Enterprise  Solutions  Division,  Bangalore,  India 

V 

Wipro  Technologies,  Global  R  &  D  (formerly  Technology  Solutions),  Bangalore, 
India 

v 

1 


V 

1 
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Appendix  C:  SEI  Strategy  for  Ensuring 

Valid  Implementation  and 
Appraisal  of  Level  4  and  5 
Process  Areas:  October  28, 
1999 


Introduction  and  Background 

When  the  Capability  Maturity  Model®  (CMM®)  for  Software  (SW-CMM)  was  initially  pub- 
lished  by  the  Software  Engineering  Institute  (SEI),  comparatively  little  was  known  about  the 
practices  of  organizations  at  levels  4  and  5.  Although  conceptually  it  was  understood  what  lev¬ 
els  4  and  5  should  look  like,  real  examples  of  these  types  of  organizations  were  scarce  in  the 
software  community.  The  concepts  for  the  level  4  and  5  key  process  areas  were  based  on  ideas 
of  some  of  the  leading  thinkers  in  the  software  community.  The  goals  and  practices  for  the  key 
process  areas  were  adapted  from  other  non-software  industries.  These  goals  and  practices  were 
tailored  to  what  was  believed  to  be  reasonable  for  a  software  process. 

While  initially  proposed  level  4  practices  were  based  on  statistical  quality  control  (SW-CMM 
V1.0),  the  current  SW-CMM  V  1.1  emphasizes  quantitative  management  arid  quantitative  con¬ 
trol.  This  more  accurately  reflected  the  state-of-the-practice  in  software  engineering  at  the  time. 
Over  the  last  few  years,  the  mature  software  organizations  have  used  rigorous  statistics  and  sta¬ 
tistical  process  control  (SPC)  techniques  in  their  implementation  of  maturity  levels  4  and  5. 
Based  on  these  observations  and  other  sources,  the  level  4  and  5  key  process  areas  of  SW- 
CMM  V2.0  draft  C  were  focused  toward  the  use  of  rigorous  statistical  methods.  The  reviewers 
of  the  drafts  of  SW-CMM  V2.0  generally  supported  this  approach.  CMMI  models  are  to  a  large 
degree  adopting  the  level  4  and  5  key  process  areas  from  SW-CMM  V2.0  draft  C. 

As  the  number  of  organizations  assessed  at  levels  4  and  5  has  grown  over  the  last  few  years, 
there  is  a  concern  that  there  is  a  wide  range  of  interpretations  of  the  level  4  and  5  key  process 
areas.  There  may  be  interpretations  that  are  inconsistent  with  what  was  intended  by  the  goals 
and  practices  of  these  key  process  areas.  This  potential  has  become  a  serious  concern  to  many 
users  of  the  SW-CMM.  As  more  and  more  maturity  level  4  and  5  appraisals  occur  and  are  de¬ 
scribed  in  papers  and  presentations,  the  essential  requirements  for  achieving  level  4  or  level  5 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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may  be  eroding.  While  there  are  likely  to  be  differences  of  interpretation  over  a  process  model 
such  as  the  SW-CMM,  a  common  understanding  must  be  developed  in  the  software  community 
as  to  what  constitutes  an  acceptable  approach  to  maturity  levels  4  and  5,  including  both  the  de¬ 
ployment  and  institutionalization.  If  this  potential  erosion  is  not  stopped  and  reversed,  the  use¬ 
fulness  of  the  SW-CMM,  as  well  as  other  CMMs,  will  be  in  serious  jjeopardy. 

Purpose 

The  purpose  in  establishing  this  strategy  is  to  define  specific  actions  that  the  SEI  and  others 
in  the  software  community  can  implement  so  that  users  of  the  SW-CMM  V 1 . 1  and  other  ca¬ 
pability  maturity  models  have  a  clear  and  common  understanding  of  what  needs  to  be  done  to 
implement  maturity  levels  4  and  5.  In  this  regard,  we  need  to  address  the  needs  of  (1)  process 
improvement  practitioners,  (2)  process  appraisal  teams,  and  (3)  management  of  the  organiza¬ 
tions  that  are  the  recipients  of  the  process  improvement  and  appraisals.  With  regard  to  the 
appraisal  teams,  we  also  need  to  define  actions  to  ensure  that  the  appraisals  of  the  maturity 
level  4  and  5  key  process  areas  are  valid  and  consistent  across  appraisal  teams. 

Candidate  Strategies  for  Addressing  the  Need 

In  addressing  the  concern  of  inconsistent  and  inappropriate  interpretation  of  the  level  4  and  5 
key  process  areas,  our  strategy  needs  to  approach  the  problem  from  multiple  directions. 

•  It  has  to  address  both  honest  misunderstanding  as  well  as  any  intentional  misapplication. 

•  It  has  to  undo  some  of  the  damage  already  caused  as  well  as  prevent  future  erosion. 

•  It  has  to  provide  support  for  organizations  to  achieve  satisfaction  against  SW-CMM  V  1.1 
as  well  as  support  organizations  that  want  to  continue  to  evolve  beyond  SW-CMM  V 1 . 1 . 

•  It  has  to  start  by  gaining  a  common  understanding  and  approach  within  the  SEI  and  then 
transition  these  ideas  into  the  rest  of  the  CMM  community. 

The  following  is  a  list  of  candidate  actions  that  can  be  taken,  as  well  as  actions  completed  or 
underway.  The  completed  and  underway  actions  are  included  for  completeness  and  to  ensure 
that  they  are  considered  in  the  overall  strategy. 

1.  Internal  SEI  Actions3 

a.  Establish  and  maintain  the  plan  for  implementing  this  strategy  to  address  the  problem 
of  inconsistent  and  inappropriate  interpretation  of  the  level  4  and  5  key  process  areas. 

b.  Define  the  steps  that  will  be  taken  to  ensure  consistency  and  validity  in  appraisals  of 
high  maturity  organizations. 

c.  Develop  SEI  level  4  and  5  qualified  individuals  to  represent  SEI  in  level  4  and  5  dis¬ 
cussions,  training,  consultation,  and  appraisals. 


3  These  internal  actions  are  intended  to  support  or  provide  the  basis  for  external  actions. 
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2. 


Define  Improvements  for  Maturity  Level  4  and  5 
Process  Areas  and  Appraisals  of  SW-CMM  VI. 1  and 
other  CMMs. 

a.  Establish  clear  criteria  for  determining  satisfaction  of  the  level  4  and  5  key  process 
areas. 

b.  Define  heuristics  for  appraising  an  organization  against  the  level  4  and  5  key  process 
areas. 

c.  Develop  guidance  for  crafting  process  improvement  recommendations  to  enable  or¬ 
ganizations  to  achieve  level  4  and  5. 

d.  Develop  guidance  for  crafting  process  improvement  recommendations  for  organiza¬ 
tions  that  achieved  level  4  and  5  and  want  to  improve  beyond  SW-CMM  VI.  1  and 
other  CMMs. 

3.  CMM  Training  and  Education 

a.  Offer  the  “Software  Engineering  and  Management  Practices  of  High  Maturity  Or¬ 
ganizations”.  Scheduled  dates:  February  8-10,  June  20-22,  October  3-5,  2000. 

b.  Offer  the  “Statistical  Process  Control  for  Software”.  Scheduled  dates:  February  1-3, 
May  2-4,  October  10-12,  2000. 

c.  Develop  and  offer  Intermediate  Model  Training  for  potential  Lead  Assessors  and 
SEPG  persons  who  need  more  in-depth  knowledge  of  the  CMM  than  provided  in  the 
“Intro”  course,  including  more  robust  treatment  of  process  areas  and  real-world  ex¬ 
amples  of  practices  at  each  maturity  level. 

4.  Appraisal  Method  Training  and  Education 

a.  Upgrade  the  “CBA  Lead  Assessor”  course  for  more  emphasis  on  high  maturity  prac- 
tices. 

b.  Upgrade  the  “SCE  Lead  Evaluator”  course  for  more  emphasis  on  high  maturity  prac¬ 
tices. 

c.  Require  Intermediate  Model  Training  for  experienced  Lead  Assessors  and  Lead 
Evaluators. 

d.  Provide  training  for  existing  Lead  Assessors  and  Lead  Evaluators  on  High  Maturity 
Practices. 

5.  Workshops 

a.  Conduct  CMM  High  Maturity  Practices  workshops.  Scheduled  for  November  16-18, 
1999. 

6.  Conference  and  Journal  Papers 

a.  Provide  qualified  referees  to  conference  program  committees  and  journal  editorial 
boards  to  review  papers  dealing  with  level  4  and  5  issues. 

b.  Publish  and  present  a  paper:  “Building  and  Assessing  High  Maturity  Organizations” 
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7.  Technical  Reports  and  Publications 

a.  “Practical  Software  Measurement:  Measuring  for  Process  Management  and  Im¬ 
provement”  handbook:  CMU/SEI-97-HB-003. 

b.  “Statistical  Process  Control  for  Software”  by  William  Florae  and  Anita  Carleton, 
published  by  Addison  Wesley,  1999. 

8.  Informal  Communications  Between  the  SEI  and  the 
CMM  Community 

a.  Publish  white  papers  on  CMM  interpretation  and  appraisal  issues 

b.  Publish  questions  and  answers  on  SW-CMM  interpretation  and  appraisal  issues 

c.  Use  the  Lead  Assessor  Web  Center  as  a  forum  for  discussion  of  these  issues  and  re¬ 
view  comments  on  papers. 
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